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Abstract. The algorithmic theory of randomness is well developed when the underlying 
space is the set of finite or infinite sequences and the underlying probability distribution 
is the uniform distribution or a computable distribution. These restrictions seem artificial. 
Some progress has been made to extend the theory to arbitrary Bernoulli distributions (by 
Martin-L6f), and to arbitrary distributions (by Levin). We recall the main ideas and problems 
Oh' of Levin's theory and report further progress in the same framework. The issues are the 

following: 

- Allow non-compact spaces (like the space of continuous functions, underlying the Brown- 
ed ian motion) . 

CN| , - The uniform test (deficiency of randomness) d P (x) (depending both on the outcome x 

and the measure P) should be defined in a general and natural way. 

- See which of the old results survive: existence of universal tests, conservation of ran- 
domness, expression of tests in terms of description complexity, existence of a universal 
measure, expression of mutual information as "deficiency of independence". 

^ | - The negative of the new randomness test is shown to be a generalization of complexity in 

O ■ continuous spaces; we show that the addition theorem survives. 

The paper's main contribution is introducing an appropriate framework for studying these 
questions and related ones (like statistics for a general family of distributions) . 

> 
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1. Introduction 

1.1. Problem statement. The algorithmic theory of randomness is well developed when 
the underlying space is the set of finite or infinite sequences and the underlying probability 
distribution is the uniform distribution or a computable distribution. These restrictions 
seem artificial. Some progress has been made to extend the theory to arbitrary Bernoulli 
distributions by Martin-L6f in [15], and to arbitrary distributions, by Levin in [11, 12, 13]. 
The paper [10] by Herding and Weihrauch also works in general spaces, but it is restricted 
to computable measures. Similarly, Asarin's thesis [1] defines randomness for sample paths 
of the Brownian motion: a fixed random process with computable distribution. 

The present paper has been inspired mainly by Levin's early paper [12] (and the much 
more elaborate [13] that uses different definitions): let us summarize part of the content 
of [12]. The notion of a constructive topological space X and the space of measures over 
X is introduced. Then the paper defines the notion of a uniform test. Each test is a lower 
semicomputable function (/i, x) i— ► f^(x), satisfying J f fl (x)/j,(dx) ^ 1 for each measure 
There are also some additional conditions. The main claims are the following. 

(a) There is a universal test t M (x), a test such that for each other test / there is a constant 
c > with fn(x) < c-t At (x). The deficiency of randomness is defined as d M (x) = logt^x). 
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(b) The universal test has some strong properties of "randomness conservation": these say, 
essentially, that a computable mapping or a computable randomized transition does not 
decrease randomness. 

(c) There is a measure M with the property that for every outcome x we have t^Or) ^ !• 
In the present paper, we will call such measures neutral. 

(d) Semimeasures (semi-additive measures) are introduced and it is shown that there is 
a lower semicomputable semimeasure that is neutral (so we can assume that the M 
introduced above is lower semicomputable) . 

(e) Mutual information I(x : y) is defined with the help of (an appropriate version of) Kol- 
mogorov complexity, between outcomes x and y. It is shown that I(x : y) is essentially 
equal to dMxAi(x,y)- This interprets mutual information as a kind of "deficiency of 
independence". 

This impressive theory leaves a number of issues unresolved: 

1. The space of outcomes is restricted to be a compact topological space, moreover, a par- 
ticular compact space: the set of sequences over a finite alphabet (or, implicitly in [13], a 
compactified infinite alphabet) . However, a good deal of modern probability theory hap- 
pens over spaces that are not even locally compact: for example, in case of the Brownian 
motion, over the space of continuous functions. 

2. The definition of a uniform randomness test includes some conditions (different ones 
in [12] and in [13]) that seem somewhat arbitrary. 

3. No simple expression is known for the general universal test in terms of description 
complexity. Such expressions are nice to have if they are available. 

1.2. Content of the paper. The present paper intends to carry out as much of Levin's 
program as seems possible after removing the restrictions. It leaves a number of questions 
open, but we feel that they are worth to be at least formulated. A fairly large part of the 
paper is devoted to the necessary conceptual machinery. Eventually, this will also allow 
to carry further some other initiatives started in the works [15] and [11]: the study of 
tests that test nonrandomness with respect to a whole class of measures (like the Bernoulli 
measures). 

Constructive analysis has been developed by several authors, converging approximately 
on the same concepts. We will make use of a simplified version of the theory introduced 
in [24]. As we have not found a constructive measure theory in the literature fitting our 
purposes, we will develop this theory here, over (constructive) complete separable metric 
spaces. This generality is well supported by standard results in measure theoretical proba- 
bility, and is sufficient for constructivizing a large part of current probability theory. 

The appendix recalls some of the needed topology, measure theory and constructive anal- 
ysis. Constructive measure theory is introduced in Section 2. 

Section 3 introduces uniform randomness tests. It proves the existence of universal uni- 
form tests, under a reasonable assumption about the topology ("recognizable Boolean in- 
clusions"). Then it proves conservation of randomness. 

Section 4 explores the relation between description (Kolmogorov) complexity and uni- 
form randomness tests. After extending randomness tests over non-normalized measures, 
its negative logarithm will be seen as a generalized description complexity. 

The rest of the section explores the extent to which the old results characterizing a ran- 
dom infinite string by the description complexity of its segments can be extended to the 
new setting. We will see that the simple formula working for computable measures over 
infinite sequences does not generalize. However, still rather simple formulas are available 
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in some cases: namely, the discrete case with general measures, and a space allowing a 
certain natural cell decomposition, in case of computable measures. 

Section 5 proves Levin's theorem about the existence of a neutral measure, for compact 
spaces. Then it shows that the result does not generalize to non-compact spaces, not even 
to the discrete space. It also shows that with our definition of tests, the neutral measure 
cannot be chosen semicomputable, even in the case of the discrete space with one-point 
compactification. 

Section 6 takes up the idea of viewing the negative logarithm of a randomness test as 
generalized description complexity. Calling this notion algorithmic entropy, this section ex- 
plores its information-theoretical properties. The main result is a (nontrivial) generalization 
of the addition theorem of prefix complexity (and, of course, classical entropy) to the new 
setting. 

1.3. Some history. Attempts to define randomness rigorously have a long but rather sparse 
history starting with von Mises and continuing with Wald, Church, Ville. Kolmogorov's work 
in this area inspired Martin-L6f whose paper [15] introduces the notion of randomness used 
here. 

Description complexity has been introduced independently by Solomonoff, Kolmogorov 
and Chaitin. Prefix complexity has been introduced independently by Levin and Chaitin. 
See [14] for a discussion of priorities and contributions. The addition theorem (whose 
generalization is given here) has been proved first for Kolmogorov complexity, with a log- 
arithmic error term, by Kolmogorov and Levin. For the prefix complexity its present form 
has been proved jointly by Levin and Gacs in [6], and independently by Chaitin in [5] . 

In his PhD thesis, Martin-L6f also characterized randomness of finite sequences via their 
complexity. For infinite sequences, complete characterizations of their randomness via the 
complexity of their segments were given by Levin in [11], by Schnorr in [17] and in [5] 
(attributed). Of these, only Levin's result is formulated for general computable measures: 
the others apply only to coin-tossing. Each of these works uses a different variant of de- 
scription complexity. Levin uses monotone complexity and the logarithm of the universal 
semicomputable measure (see [8] for the difficult proof that these two complexities are 
different) . Schnorr uses "process complexity" (similar to monotone complexity) and prefix 
complexity. The work [7] by the present author gives characterizations using the original 
Kolmogorov complexity (for general computable measures) . 

Uniform tests over the space of infinite sequences, randomness conservation and neutral 
measures were introduced in Levin's work [12]. The present author could not verify every 
result in that paper (which contains no proofs) ; he reproduced most of them with a changed 
definition in [7]. A universal uniform test with yet another definiton appeared in [13]. In 
this latter work, "information conservation" is a central tool used to derive several results in 
logic. In the constellation of Levin's concepts, information conservation becomes a special 
case of randomness conservation. We have not been able to reproduce this exact relation 
with our definition here. 

The work [9] is based on the observation that Zurek's idea on "physical" entropy and 
the "cell volume" approach of physicists to the definition of entropy can be unified: Zurek's 
entropy can be seen as an approximation of the limit arising in a characterization of a 
randomness test by complexity. The author discovered in this same paper that the negative 
logarithm of a general randomness test can be seen as a generalization of complexity. He 
felt encouraged by the discovery of the generalized addition theorem presented here. 

The appearence of other papers in the meantime (including [10]) convinced the au- 
thor that there is no accessible and detailed reference work on algorithmic randomness 
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for general measures and general spaces, and a paper like the present one, developing the 
foundations, is needed. (Asarin's thesis [1] does develop the theory of randomness for the 
Brownian motion. It is a step in our direction in the sense that the space is not compact, 
but it is all done for a single explicitly given computable measure.) 

We do not advocate the uniform randomness test proposed here as necessarily the "defin- 
itive" test concept. Perhaps a good argument can be found for some additional conditions, 
similar to the ones introduced by Levin, providing additional structure (like a semicom- 
putable neutral measure) while preserving naturalness and the attractive properties pre- 
sented here. 

1.4. Notation for the paper. (Nothing to do with the formal concept of "notation", in- 
troduced later in the section on constructive analysis.) The sets of natural numbers, inte- 
gers, rational numbers, real numbers and complex numbers will be denoted respectively 
by N, Z, Q, R. The set of nonnegative real numbers will be denoted by R + . The set of real 
numbers with — oo,oo added (with the appropriate topology making it compact) will be 
denoted by R. We use A and V to denote min and max, further 

\x\ + = x V 0, \x\~ = | — x\ + 

for real numbers x. We partially follow [24], [3] and [10]. In particular, adopting the 
notation of [24] , we denote intervals of the real line as follows, (to avoid the conflict with 
the notation of a pair (a, b) of objects). 

[a; b] = { x : a ^ x ^ b }, (a; b) = { x : a < x < b }, [a; b) = { x : a ^ x < b }. 

If X is a set then X* is the set of all finite strings made up of elements of X, including the 
"empty string" A. We denote by X u the set of all infinite sequences of elements of X. If A 
is a set then 1^0*0 is its indicator function, defined to be 1 if x e A and to otherwise. For 
a string x, its length is \x\, and 

x^ = (x(l),...,x(n)). 

The relations 

f<9, f<9 

mean inequality to within an additive constant and multiplicative constant respectively. The 
first is equivalent to / ^ g + 0(1), the second to / = O(g). The relation / = g means f < g 
and f > g. 

Borrowing from [16], for a function / and a measure fx, we will use the notation 

m/ = J f(x)Kdx), n y f(x,y) = J f{x,y)K d y)- 

2. Constructive measure theory 

The basic concepts and results of measure theory are recalled in Section B. For the 
theory of measures over metric spaces, see Subsection B.6. We introduce a certain fixed, 
enumerated sequence of Lipschitz functions that will be used frequently. Let JF be the set 
of functions of the form g u>r> i/ ri where u G D, r G Q, n = 1, 2, . . . , and 

9u,r,e(x) = |1 - \d(x,u) - r\ + / e\ + 

is a continuous function that is 1 in the ball B(u,r), it is outside B(u,r + e), and takes 
intermediate values in between. Let 



£ = {91,92, ■■■} 



(2.1) 
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be the smallest set of functions containing JF and the constant 1, and closed under V, A 
and rational linear combinations. The following construction will prove useful later. 

Proposition 2.1. All bounded continuous functions can be obtained as the limit of an increas- 
ing sequence of functions from the enumerated countable set £ of bounded computable Lipschitz 
functions introduced in (2.1). 

The proof is routine. 

2.1. Space of measures. Let X = (X, d, D,a) be a computable metric space. In Subsec- 
tion B.6, the space M(X.) of measures over X is defined, along with a natural enumeration 
v = vm for a subbase a = a_M of the weak topology. This is a constructive topological 
space M which can be metrized by introducing, as in B.6. 2, the Prokhorov distance p(p, u): 
the infimum of all those e for which, for all Borel sets A we have p(A) ^ v(A £ ) + e, where 
A £ = { x : 3y G A d(x, y) < e }. Let Dm be the set of those probability measures that are 
concentrated on finitely many points of D and assign rational values to them. Let ojm be a 
natural enumeration of Dm- Then 

(M,p,D M ,a M ) (2.2) 

is a computable metric space whose constructive topology is equivalent to M. Let U = 
B(x, r) be one of the balls in X, where x & Dx, r G Q. The function p ^ p(U) is typically 
not computable, not even continuous. For example, if X = E and U is the open interval 
(0; 1), the sequence of probability measures 5i/ n (concentrated on 1/n) converges to 5q, but 
Si/ n (U) = 1, and 5q(U) = 0. The following theorem shows that the situation is better with 
p,\—>p,f for computable /: 

Proposition 2.2. Let X = (X, d, D, a) be a computable metric space, and let M = 
(y\4(X), cr, v) be the effective topological space of probability measures over X. If function 
f : X — > R is bounded and computable then fj,\—>pfis computable. 

Proof sketch. To prove the theorem for bounded Lipschitz functions, we can invoke the 
Strassen coupling theorem B.16. 

The function / can be obtained as a limit of a computable monotone increasing sequence 
of computable Lipschitz functions />, and also as a limit of a computable monotone de- 
creasing sequence of computable Lipschitz functions /< . In step n of our computation of 
[if, we can approximate [if> from above to within 1/n, and p,f< from below to within 
1/n. Let these bounds be a> and a^. To approximate pf to within e, find a stage n with 
a> - a< + 2/n < e. □ 

2.2. Computable measures and random transitions. A measure p is called computable if 
it is a computable element of the space of measures. Let {gi} be the set of bounded Lipschitz 
functions over X introduced in (2.1). 

Proposition 2.3. Measure \i is computable if and only if so is the function i i— > pgi. 

Proof. The "only if" part follows from Proposition 2.2. For the "if" part, note that in order to 
trap p within some Prokhorov neighborhood of size e, it is sufficient to compute pgi within 
a small enough 5, for alii < n for a large enough n. □ 

Example 2.4. Let our probability space be the set R of real numbers with its standard topol- 
ogy. Let a < b be two computable real numbers. Let p be the probability distribution 
with density function f(x) = -^l[ ai b]( x ) (the uniform distribution over the interval [a; 6]). 
Function f(x) is not computable, since it is not even continuous. However, the measure p is 
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computable: indeed, pg^ = J gi(x)dx is a computable sequence, hence Proposition 2.3 
implies that p is computable. <0 

The following theorem compensates somewhat for the fact mentioned earlier, that the 
function p i— > p(U) is generally not computable. 

Proposition 2.5. Let p be a finite computable measure. Then there is a computable map h 
with the property that for every bounded computable function f with \ f\ ^ 1 with the property 
p(f" 1 (0)) = 0, ifw is the name of f then h(w) is the name of a program computing the value 

p{x : f(x) < 0}. 

Proof. Straightforward. □ 

Remark 2.6. Suppose that there is a computable function that for each i computes a Cauchy 
sequence j i— > rrii{j) with the property that for i < j± < j% we have \m,i(ji) — mi{j<i)\ < 2~ jl , 
and that for all n, there is a measure u with the property that for all i < n, vgi = mj(n). 
Is there a measure p, with the property that for each i we have limy mj(j) = /^j? Not 
necessarily, if the space is not compact. For example, let X = {1, 2, 3, ... } with the discrete 
topology. The sequences rrii(j) = for j > i satisfy these conditions, but they converge 
to the measure 0, not to a probability measure. To guarantee that the sequences m;(j) 
indeed define a probability measure, progress must be made, for example, in terms of the 
narrowing of Prokhorov neighborhoods. 

Let now X, Y be computable metric spaces. They give rise to measurable spaces with 
(T-algebras A, B respectively. Let A = {\ x : x £ X} be a. probability kernel from X to Y 
(as defined in Subsection B.5). Let {^} be the set of bounded Lipschitz functions over Y 
introduced in (2.1). To each gi, the kernel assigns a (bounded) measurable function 

fi(x) = (Agi)(x) = \ y x gi{y). 

We will call A computable if so is the assignment (i, x) i— > fi(x). In this case, of course, each 
function fi(x) is continuous. The measure A*p is determined by the values A*gi = p(Ag{), 
which are computable from (i, p) and so the function p, i— > A*p is computable. 

Example 2.7. A computable function h : X — > Y defines an operator A/ t with Ahg = g o h 
(as in Example B.12). This is a deterministic computable transition, in which fi(x) = 

(Ahgi)(x) = gi(h(x)) is, of course, computable from (i,x). We define h*fi = A* h fi. 

2.3. Cells. As pointed out earlier, it is not convenient to define a measure p constructively 
starting from p(T) for open cells T. The reason is that no matter how we fix T, the func- 
tion n i— > p(T) is typically not computable. It is better to work with bounded computable 
functions, since for such a function /, the correspondence n \- ► pf is computable. 

Under some special conditions, we will still get "sharp" cells. Let / be a bounded com- 
putable function over X, let a\ < ■ ■ ■ < be rational numbers, and let p be a computable 
measure with the property that / u/~ 1 (a J ) = for all j. In this case, we will say that a.j 
are regular points of / with respect to p. Let qo = — oo, a^+i = oo, and for j = 0, . . . , k, 
let Let Uj = j + 1)). The sequence of disjoint r.e. open sets (Uq, . . . , C4) will be 

called the partition generated by f, cti, . . . , a^. (Note that this sequence is not a partition in 
the sense of |J • Uj = X, since the boundaries of the sets are left out.) If we have several 
partitions (Uio, . . . , Ui s k), generated by different functions fi {i = 1, . . . , m) and different 
regular cutoff sequences (a^- : j = 1, . . . , ki), then we can form a new partition generated 
by all possible intersections 

Vn,..., jn = u ltjl n---nu m)jm . 
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A partition of this kind will be called a regular partition. The sets j n will be called the 
cells of this partition. 

Proposition 2.8. In a regular partition as given above, the values (iVj u ... i j n are computable 
from the names of the functions fa and the cutoff points cqj. 

Proof. Straightforward. □ 

Assume that a computable sequence of functions b\ (x), 62(^)1 • • • over X is given, with 
the property that for every pair x±,X2 € X with x\ ^ x%, there is a j with bj(xi) ■ bj(x2) < 

0. Such a sequence will be called a separating sequence. Let us give the correspondence 
between the set B w of infinite binary sequences and elements of the set 

X° = {x€X:b j (x)^0, j = l,2,...}. 
For a binary string s± ■ ■ ■ s n = s 6 B*, let 

r, 

be the set of elements of X with the property that for j = 1, . . . , n, if sj = then bj(u) < 0, 
otherwise bj(u) > 0. This correspondence has the following properties. 

(a) T A = X. 

(b) For each s e B, the sets T s q and r a i are disjoint and their union is contained in T s . 

(c) For x G X°, we have {x} = fixers ^s- 

If s has length n then T s will be called a canonical n-cell, or simply canonical cell, or n-cell. 
From now on, whenever T denotes a subset of X, it means a canonical cell. We will also 
use the notation 

l(T s ) = l(s). 

The three properties above say that if we restrict ourselves to the set X° then the canonical 
cells behave somewhat like binary subintervals: they divide X° in half, then each half again 
in half, etc. Moreover, around each point, these canonical cells become "arbitrarily small", 
in some sense (though, they may not be a basis of neighborhoods). It is easy to see that 
if r si , T S2 are two canonical cells then they either are disjoint or one of them contains the 
other. If T Sl c T S2 then S2 is a prefix of s\. If, for a moment, we write r° = T s n X° then 
we have the disjoint union T° = r° U T^. For an n-element binary string s, for x e T s , we 
will write 

H(s) = /i(r s ). 

Thus, for elements of X°, we can talk about the n-th bit x n of the description of x: it 
is uniquely determined. The 2 n cells (some of them possibly empty) of the form T s for 
l{s) = n form a partition 

0fX°. 

Examples 2.9. 

1. If X is the set of infinite binary sequences with its usual topology, the functions b n (x) = 
x n — 1/2 generate the usual cells, and X° = X. 

2. If X is the interval [0; 1], let b n (x) = — sin(2 n 7rx). Then cells are open intervals of the 
form (k-2~ n ; (k+l)-2 n ), the correspondence between infinite binary strings and elements 
of X° is just the usual representation of x as the binary decimal string O.X1X2 
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When we fix canonical cells, we will generally assume that the partition chosen is also 
"natural". The bits xi,X2,-- - could contain information about the point x in decreasing 
order of importance from a macroscopic point of view. For example, for a container of 
gas, the first few bits may describe, to a reasonable degree of precision, the amount of 
gas in the left half of the container, the next few bits may describe the amounts in each 
quarter, the next few bits may describe the temperature in each half, the next few bits may 
describe again the amount of gas in each half, but now to more precision, etc. From now 
on, whenever T denotes a subset of X, it means a canonical cell. From now on, for elements 
of X°, we can talk about the n-th bit x n of the description of x: it is uniquely determined. 

The following observation will prove useful. 

Proposition 2.10. Suppose that the space X is compact and we have a separating sequence 
bi (x) as given above. Then the cells T s form a basis of the space X. 

Proof. We need to prove that for every ball B(x, r), there is a cell x G T s c B(x, r). Let C 
be the complement of B(x, r). For each point y of C, there is an i such that 6j(x) • bi(y) < 0. 
In this case, let J° = { z : bi(z) < }, J 1 = { z : bi{z) > }. Let J(y) = J' p such that y G J' p . 
Then C C [j J(y), and compactness implies that there is a finite sequence y\, . . . , y^ with 

c c UjLi J (Vj)- Clearly there is a cell x G T s C B(x, r) \ {f. =x J(y j ). □ 

3. Uniform tests 

3.1. Universal uniform test. Let X = (X,d,D,a) be a computable metric space, and let 
M = (.A/f (X), a, v) be the constructive topological space of probability measures over X. A 
randomness test is a function / : M x X — » R with the following two properties. 

Condition 3.1. 

1. The function i— > is lower semicomputable. (Then for each p,, the integral 

fJ>U = ^ x f^{x) exists.) 

2. /u/m < 1- 



The value f^x) is intended to quantify the nonrandomness of the outcome x with respect 
to the probability measure fi. The larger the values the less random is x. Condition 3.1.2 
guarantees that the probability of those outcomes whose randomness is ^ m is at most 
1/m. The definition of tests is in the spirit of Martin-L6f 's tests. The important difference is 
in the semicomputability condition: instead of restricting the measure p to be computable, 
we require the test to be lower semicomputable also in its argument p. 

Just as with Martin-Lof's tests, we want to find a universal test; however, we seem to 
need a condition on the space X. Let us say that a sequence i >—>■ Ui of sets has recognizable 
Boolean inclusions if the set 

{ (S, T):S,T are finite, |~| Ui C {JUj} 

is recursively enumerable. We will say that a computable metric space has recognizable 
Boolean inclusions if this is true of the enumerated basis consisting of balls of the form 
B(x, r) where x G D and r > is a rational number. 

It is our conviction that the important metric spaces studied in probability theory have 
recognizable Boolean inclusions, and that proving this in each individual case should not 
be too difficult. For example, it does not seem difficult to prove this for the space C[0; 1] of 
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Example C.6, with the set of rational piecewise-linear functions chosen as D. But, we have 
not carried out any of this work! 

Theorem 1. Suppose that the metric space X has recognizable Boolean inclusions. Then there 
is a universal test, that is a test t M (x) with the property that for every other test /^(x) there is 
a constant c/ > with cff^x) < t M (x). 

Proof. 

1 . We will show that there is a mapping that to each name u of a lower semicomputable 
function (/i, x) i— > g(fi,x) assigns the name of a lower semicomputable function g'(fi,x) 
such that fi x g'(fi, x) ^ 1, and if g is a test then g' = g. 

To prove the statement, let us represent the space M rather as 

M = (M(X),p,D,a M ), (3.1) 

as in (2.2). Since g{p,,x) is lower semicomputable, there is a computable sequence of 
basis elements Ui C M and V% C X and rational bounds r; such that 

g(H,x) = sup rjl^^ly^a;). 

i 

Let h n (p,,x) = maxjsjnrjlf/^^ly^x). Let us also set ho(p,,x) = 0. Our goal is to show 
that the condition V/i fj, x h n (fi, x) ^ 1 is decidable. If this is the case then we will be done. 
Indeed, we can define h' n (p,,x) recursively as follows. Let h' (p,,x) = 0. Assume that 
h' n (fi,x) has been defined already. If V/x /j, x h n+ i(/j,,x) ^ 1 then = h n+ i(fj,,x); 

otherwise, it is h' n (fi,x). The function g'(p,x) = sup n h' n (p,,x) clearly satisfies our re- 
quirements. 

We proceed to prove the decidability of the condition 

V/i fi x hn(ji, x) < 1. (3.2) 

The basis elements tyj can be taken as balls B(qi, 5i) for a computable sequence qi £ D 
and computable sequence of rational numbers 5i > 0. Similarly, the basis element Ui is 
the set of measures that is a ball B{a^ £j), in the metric space (3.1). Here, using nota- 
tion (B.7), <Ji is a measure concentrated on a finite set S^. According to Proposition B.17, 
the ball Ui is the set of measures p, satisfying the inequalities 

/j(A^) > ai{A)-ei 

for all A cz Si. For each n, consider the finite set of balls 

B n = { B( qi , Si) : i < n } U { B(s, : i < n, s G }. 

Consider all sets of the form 

^a,b = H U \ |J L7 
U&A UeB 

for all pairs of sets A,B c S n . These sets are all finite intersections of balls or com- 
plements of balls from the finite set B n of balls. The space X has recognizable Boolean 
inclusions, so it is decidable which of these sets Ua,b are nonempty. The condition (3.2) 
can be formulated as a Boolean formula involving linear inequalities with rational coef- 
ficients, for the variables ^a,b = P>(Ua,b), for those A, B with Ua,b 7^ 0. The solvability 
of such a Boolean condition can always be decided. 

2. Let us enumerate all lower semicomputable functions g u (p,, x) for all the names u. With- 
out loss of generality, assume these names to be natural numbers, and form the functions 
g' u (fi,x) according to the assertion 1 above. The function t = ^2 u 2~ u ~ 1 g' u will be the 
desired universal test. 
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□ 

From now on, when referring to randomness tests, we will always assume that our space 
X has recognizable Boolean inclusions and hence has a universal test. We fix a universal 
test t^x), and call the function 

d M (x) = logt M (x). 

the deficiency of randomness of x with respect to /i. We call an element i£l random with 
respect to [i if d M (x) < oo. 

Remark 3.2. Tests can be generalized to include an arbitrary parameter y: we can talk about 
the universal test 

t M | y), 

where y comes from some constructive topological space Y. This is a maximal (within 
a multiplicative constant) lower semicomputable function (x,y,fx) i— > f(x,y,n) with the 
property [i x f(x, y, fi) ^ 1. 

3.2. Conservation of randomness. For i = 1,0, let Xj = (Xi, di, Di, cti) be computable 
metric spaces, and let M, = (M(X.i), <Ji, Vj) be the effective topological space of probability 
measures over Xj. Let A be a computable probability kernel from Xi to Xo as defined 
in Subsection 2.2. In the following theorem, the same notation d^(x) will refer to the 
deficiency of randomness with respect to two different spaces, Xi and Xo, but this should 
not cause confusion. Let us first spell out the conservation theorem before interpreting it. 

Theorem 2. For a computable probability kernel A from Xi to Xo, we have 

A£t A * M (j/) < t M (x). (3.3) 

Proof. Let t v (x) be the universal test over Xo. The left-hand side of (3.3) can be written as 

Ufj, = At A >. 

According to (B.4), we have [iu^ = (A*/x)tA* Al which is ^ 1 since t is a test. If we show that 
(fx, x) I— > u fl (x) is lower semicomputable then the universality of t M will imply < t M . 

According to Proposition C.7, as a lower semicomputable function, t u (y) can be written 
as sup n g n (u,y), where (g n (y,y)) is a computable sequence of computable functions. We 
pointed out in Subsection 2.2 that the function p, i— > A*// is computable. Therefore the 
function (n,fi,x) i— > g n (A* /j, f(x)) is also a computable. So, u fl (x) is the supremum of a 
computable sequence of computable functions and as such, lower semicomputable. □ 

It is easier to interpret the theorem first in the special case when A = A^ for a computable 
function h : X\ — > Ao, as in Example 2.7. Then the theorem simplifies to the following. 

Corollary 3.3. For a computable function h : Xi — > Xq, we have d/j* M (/i(x)) < d^(x). 

Informally, this says that if x is random with respect to /j, in Xi then h(x) is essentially at 
least as random with respect to the output distribution h*fi in Xo- Decrease in randomness 
can only be caused by complexity in the definition of the function h. It is even easier to 
interpret the theorem when /i is defined over a product space Xi x X2, and h(xi, X2) = x\ 
is the projection. The theorem then says, informally, that if the pair (x\, X2) is random with 
respect to then x\ is random with respect to the marginal ii\ = h*fi of fx. This is a very 
natural requirement: why would the throwing-away of the information about X2 affect the 
plausibility of the hypothesis that the outcome x\ arose from the distribution /ii? 
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In the general case of the theorem, concerning random transitions, we cannot bound the 
randomness of each outcome uniformly. The theorem asserts that the average nonrandom- 
ness, as measured by the universal test with respect to the output distribution, does not 

increase. In logarithmic notation: \%2 dA *^ < d M (x), or equivalently, f 2 dA * m^) X x (dy) < 
d M (x). 

Corollary 3.4. Let Abe a computable probability kernel from Xi to X . There is a constant c 
such that for every x € X 1 , and integer m > Owe have 

K{ y ■ d A >(y) > d M (x) + m + c } < 2~ m . 

Thus, in a computable random transition, the probability of an increase of randomness 
deficiency by m units (plus a constant c) is less than 2~ m . The constant c comes from the 
description complexity of the transition A. 

A randomness conservation result related to Corollary 3.3 was proved in [10]. There, the 
measure over the space Xo is not the output measure of the transformation, but is assumed 
to obey certain inequalities related to the transformation. 

4. Tests and complexity 

4.1. Description complexity. 

4.1.1. Complexity, semimeasures, algorithmic entropy. Let X = £*. For x E S* for some 
finite alphabet E, let H(x) denote the prefix-free description complexity of the finite se- 
quence x as defined, for example, in [14] (where it is denoted by K(x)). For completeness, 
we give its definition here. Let A : {0, 1}* x S* — > S* be a computable (possibly partial) 
function with the property that if A(pi,y) and A(p2,y) are defined for two different strings 
Pi,P2, then p\ is not the prefix of pi. Such a function is called a (prefix-free) interpreter. We 
denote 

H (x | y) = min \p\. 

A(p,y)=x 

One of the most important theorems of description complexity is the following: 

Proposition 4.1 (Invariance Theorem, see for example [14]). There is an optimal interpreter 
T with the above property: with it, for every interpreter A there is a constant ca with 

H T (x\y)^H A (x\y)+c A . 

We fix an optimal interpreter T and write H(x \ y) = H T (x \ y), calling it the conditional 
complexity of a string x with respect to string y. We denote H(x) = H(x \ A). Let 

m(x) = 2- H V>. 

The function m(x) is lower semicomputable with ^ x m(x) < 1. Let us call any real function 
f(x) ^ over S* with Y2 X fi x ) ^ la semimeasure. The following theorem, known as the 
Coding Theorem, is an important tool. 

Proposition 4.2 (Coding Theorem). For every lower semicomputable semimeasure f there is 
a constant c > with m(x) ^ c • f(x). 

Because of this theorem, we will say that m(x) is a universal lower semicomputable 
semimeasure. It is possible to turn m(x) into a measure, by compactifying the discrete 
space S* into 

W= £* U {oo} 
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(as in part 1 of Example A.3; this process makes sense also for a constructive discrete 
space), and setting m(oo) = 1 — YlxeT,* m (^)- The extended measure m is not quite lower 
semicomputable since the number //(£* \ {0}) is not necessarily lower semicomputable. 

Remark 4.3. A measure /i is computable over S* if and only if the function x \— > fj,(x) is 
computable for x 6 £*. This property does not imply that the number 

1 - /i(oo) = /i(S*) = fl( x ) 

is computable. 

Let us allow, for a moment, measures // that are not probability measures: they may 
not even be finite. Metric and computability can be extended to this case (see [22]), the 
universal test t M (x) can also be generalized. The Coding Theorem and other considerations 
suggest the introduction of the following notation, for an arbitrary measure \i: 

H^x) = -d^x) = -logt^x). (4.1) 

Then, with # defined as the counting measure over the discrete set S* (that is, #(<S) = 
we have 

H{x)^H # {x). 

This allows viewing H^(x) as a generalization of description complexity: we will call this 
quantity the algorithmic entropy of x relative to the measure /i. Generalization to con- 
ditional complexity is done using Remark 3.2. A reformulation of the definition of tests 
says that H^ix) is minimal (within an additive constant) among the upper semicomputable 
functions (fj,,x) i— > fn{x) with fj, x 2~f>'( x ) ^ 1. The following identity is immediate from the 
definitions: 

H^x) =Hp{x\fi). (4.2) 

4.1.2. Computable measures and complexity. It is known that for computable fx, the test 
d fl (x) can be expressed in terms of the description complexity of x (we will prove these 
expressions below). Assume that X is the (discrete) space of all binary strings. Then we 
have 

d M (x) = - log n(x) - H(x) + 0(H(fi)). (4.3) 
The meaning of this equation is the following. Due to maximality property of the semimea- 
sure m following from the Coding Theorem 4.2 above, the expression — log /i(x) is an upper 
bound (within 0(H(fi))) of the complexity H(x), and nonrandomness of x is measured by 
the difference between the complexity and this upper bound. See [26] for a first formu- 
lation of this general upper bound relation. As a simple example, consider the uniform 
distribution \i over the set of binary sequences of length n. Conditioning everything on n, 
we obtain 

d^(x | n) = n — H(x \ n), 

that is the more the description complexity H(x \ n) of a binary sequence of length n differs 
from its upper bound n the less random is x. 

Assume that X is the space of infinite binary sequences. Then equation (4.3) must be 
replaced with 

d„{x) = sup (- log M^ n ) - H(x^ n )) + 0{H{p)). (4.4) 

n 

For the coin-tossing distribution /x, this characterization has first been first proved by 
Schnorr, and published in [5]. 
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Remark 4.4. It is possible to obtain similar natural characterizations of randomness, us- 
ing some other natural definitions of description complexity. A universal semicomputable 
semimeasure over the set f2 of infinite sequences was introduced, and a complexity 
KM(x) = — \ogmn(x) defined in [26]. A so-called "monotonic complexity", Km(i) was 
introduced, using Turing machines with one-way input and output, in [11], and a closely 
related quantity called "process complexity" was introduced in [17]. These quantities can 
also be used in a characterization of randomness similar to (4.3). The nontrivial fact that 
the complexities KM and Km differ by an unbounded amount was shown in [8] . <) 

For noncomputable measures, we cannot replace 0(H(^i)) in these relations with any- 
thing finite, as shown in the following example. Therefore however attractive and simple, 
exp(— log fi(x) — H(x)) is not a universal uniform test of randomness. 

Proposition 4.5. There is a measure \x over the discrete space X of binary strings such that 
for each n, there is an x with d M (x) = n — H(n) and — log n(x) — H(x) < 0. 

Proof. Let us treat the domain of our measure /i as a set of pairs (x,y). Let x n = n , 
for n = 1,2,.... For each n, let y n be some binary string of length n with the property 
H(x n ,y n ) > n. Let n(x n ,y n ) = 2~ n . Then - log/x(x n , y n ) - H(x n ,y n ) ^ n - n = 0. On the 
other hand, let t II (x, y) be the test nonzero only on strings x of the form x n : 

i \ _ m(n) 
tfi{x n ,y) = r= - -. 

Z-^z£B n ^\ X n, Z) 

The form of the definition ensures semicomputability and we also have 

^^(x,^)^^,?/) ^ ^m(n) < 1, 

x,y n 

therefore t fl is indeed a test. Hence t^(x,y) > t fl (x,y). Taking logarithms, d /i (j; n ,y ri ) > 
n-H(n). □ 

The same example implies that it is also not an option, even over discrete sets, to replace 
the definition of uniform tests with the ad hoc formula exp(— log u>(x) — H(x)): 

Proposition 4.6. The test defined as = exp(— log ^i(x) — H(x)) over discrete spaces X 

does not obey the conservation of randomness. 

Proof. Let us use the example of Proposition 4.5. Consider the function ir : (x, y) i— > x. The 
image of the measure fi under the projection is (7r^)(x) = J2 y M^y)- Thus, (Tru-)(x n ) = 
li(x n ,y n ) = 2~ n . We have seen log / M (x n , y n ) < 0. On the other hand, 

logUn(Tr(x n ,y n )) = -log(7r/i)(x n ) - H(x n ) = n- H(n). 

Thus, the projection ir takes a random pair (x n , y n ) into an object x n that is very nonrandom 
(when randomness is measured using the tests ffj. □ 

In the example, we have the abnormal situation that a pair is random but one of its 
elements is nonrandom. Therefore even if we would not insist on universality, the test 
exp(— log/i(x) — H{x)) is unsatisfactory. 

Looking into the reasons of the nonconservation in the example, we will notice that it 
could only have happened because the test is too special. The fact that — \og{'K^){x v ) — 
H{x n ) is large should show that the pair (x n , y n ) can be enclosed into the "simple" set {x n }x 
Y of small probability; unfortunately, this observation does not reflect on — log//(x, y) — 
H(x, y) when the measure \i is non-computable (it does for computable 
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4.1.3. Expressing the uniform test in terms of complexity. It is a natural idea to modify equa- 
tion (4.3) in such a way that the complexity H(x) is replaced with H(x | p). However, 
this expression must be understood properly. The measure p (especially when it is not 
computable) cannot be described by a finite string; on the other hand, it can be described 
by infinite strings in many different ways. Clearly irrelevant information in these infinite 
strings should be ignored. The notion of representation in computable analysis (see Sub- 
section C.l) will solve the problem. An interpreter function should have the property that 
its output depends only on p and not on the sequence representing it. Recall the topological 
space M of computable measures over our space X. An interpreter A : {0, 1}* x M — > S* 
is a computable function that is prefix-free in its first argument. The complexity 

H(x | p) 

can now be defined in terms of such interpreters, noting that the Invariance Theorem holds 
as before. To define this complexity in terms of representations, let 7m be our chosen 
representation for the space M (thus, each measure p is represented via all of its Cauchy 
sequences in the Prokhorov distance). Then we can say that A is an interpreter if it is 
(id, 7M, id) -computable, that is a certain computable function B : {0, 1}* x S w — > S* realizes 
A for every p € {0, 1}*, and for every sequence z that is a 7M-name of a measure p, we 
have B(p, z) = A(p, p). 

Remark 4.7. The notion of oracle computation and reducibility in the new sense (where the 
result is required to be independent of which representation of an object is used) may be 
worth investigating in other settings as well. 

Let us mention the following easy fact: 

Proposition 4.8. If p is a computable measure then H(x \ p) = H(x). The constant in = 
depends on the description complexity of p. 

Theorem 3. If X is the discrete space S* then we have 

d M (x) = -logp(x) - H(x | p). (4.5) 

Note that in terms of the algorithmic entropy notation introduced in (4.1), this theorem 
can be expressed as 

H^x)^H{x\p)+\ogp{x). (4.6) 

Proof. In exponential notation, equation (4.5) can be written as t At (x) = m(x \ p)/p(x). Let 
us prove > first. We will show that the right-hand side of this inequality is a test, and hence 
< t At (ir). However, the right-hand side is clearly lower semicomputable in (x,p) and when 
we "integrate" it (multiply it by p(x) and sum it), its sum is ^ 1; thus, it is a test. 

Let us prove < now. The expression t fl (x)p(x) is clearly lower semicomputable in (x, p), 
and its sum is ^ 1. Hence, it is < m(a; | p). □ 

Remark 4.9. As mentioned earlier, our theory generalizes to measures that are not proba- 
bility measures. In this case, equation (4.6) has interesting relations to the quantity called 
"physical entropy" by Zurek in [25]; it justifies calling H^x) "fine-grained algorithmic 
Boltzmann entropy" by this author in [9] . <) 

For non-discrete spaces, unfortunately, we can only provide less intuitive expressions. 
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Proposition 4.10. let X = (X,d,D,a) be a complete computable metric space, and let £ 
be the enumerated set of bounded Lipschitz functions introduced in (2.1), but for the space 
M(X) x X. The uniform test of randomness t M (x) can be expressed as 

Proof. For >, we will show that the right-hand side of the inequality is a test, and hence 
< t^ix). For simplicity, we skip the notation for the enumeration of £ and treat each 
element / as its own name. Each term of the sum is clearly lower semicomputable in 
(f,x,p), hence the sum is lower semicomputable in (x,p). It remains to show that the p- 
integral of the sum is ^ 1. But, the //-integral of the generic term is < m(/ | p), and the 
sum of these terms is ^ 1 by the definition of the function m(- | ■). Thus, the sum is a test. 

For <, note that (p, x) i— > t^{x), as a lower semicomputable function, is the supremum 
of functions in £ . Denoting their differences by fi(p, x), we have t fl (x) = ^ i fi(p>, x). The 
test property implies ^ p x fi(p, x) ^ 1. Since the function (p,i) h- > p x fi(p,x) is lower 

semicomputable, this implies p x fi(p,x) < m(i \ p), and hence 

fi (p,x) < f iM ^±J^. 

It is easy to see that for each / e £ we have 

m(i | p) ^ p(f | ft), 

i-fi=f 

which leads to (4.7). □ 

Remark 4.11. If we only want the > part of the result, then £ can be replaced with any 
enumerated computable sequence of bounded computable functions. 

4.2. Infinite sequences. In this section, we get a nicer characterization of randomness 
tests in terms of complexity, in special cases. Let Mr(X) be the set of measures p with 
p(X) = R. 

Theorem 4. Let X = N w be the set of infinite sequences of natural numbers, with the product 
topology. For all computable measures p e Mr{X), for the deficiency of randomness d^(x), 
we have 

d^x)± S up{-logp(x^ n )-H(x< n )). (4.8) 

n 

Here, the constant in = depends on the computable measure p. 

We will be able to prove the > part of the statement in a more general space, and without 
assuming computability. Assume that a separating sequence b\ , 62, • • ■ is given as defined in 
Subsection 2.3, along with the set X°. For each x 6 X°, the binary sequence xi,X2,. ■ ■ has 
been defined. Let 

p(T s ) = R-^{ p(T s >) : 1(b) = l(s'), s'^s}. 

Then (s, p) 1— ► p(T s ) is lower semicomputable, and (s, p) 1— > p(T s ) is upper semicomputable. 
And, every time that the functions bi(x) form a regular partition for p, we have /l(r s ) = 
p(T s ) for all s. Let M R (X) be the set of those measures p in Mr(X) for which p(X\X°) = 
0. 
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Theorem 5. Suppose that the space X is compact. Then for all computable measures p e 
M° R (X.), for the deficiency of randomness d fl (x), the characterization (4.8) holds. 

For arbitrary measures and spaces, we can say a little less: 

Proposition 4.12. For all measures p € Mr(X), for the deficiency of randomness d M (x), we 
have 

d M (x) > sup (-log/Z(x^ n ) - H(x< n | p)). (4.9) 

n 

Proof. Consider the function 

t f \ STi i \ m ( s I *0 m(x^ n | p) m(x^ n \ p) 

U x = X, 1 r.(a) — /-p x = >. _/ < nA ^ sup _ . 

V M 1 " n ^ X ) n p{ x ^ ) 

The function (//,x) i— > / M (x) is clearly lower semicomputable and satisfies p x f fJ- (x) < 1, and 
hence 

d^(x) > log /(a?) > sup(-logp(x^ n ) - H(x^ n | p)). 

n 

□ 

Proof o/ Theorem 4. For binary sequences instead of sequences of natural numbers, the part 

> of the inequality follows directly from Proposition 4.12: indeed, look at Examples 2.9. 
For sequences of natural numbers, the proof is completely analogous. 

The proof of < reproduces the proof of Theorem 5.2 of [7]. The computability of p 
implies that t(x) = t fl (x) is lower semicomputable. Let us first replace t(x) with a rougher 
version: 

t'(x) =max{2 n : T < t M (z) }. 

Then t'{x) = t(x), and it takes only values of the form 2 n . It is also lower semicomputable. 
Let us abbreviate: 

l y (x) = l#(x), p(y) = /i(?/N w ). 
For every lower semicomputable function / over there are computable sequences yi e 
N* and ^ G Q with f(x) = sup^ rjl y . (x), with the additional property that if i < j and 
l yi (x) = l yj (x) = 1 then ri < rj. Since t'(x) only takes values of the form 2 n , there are 
computable sequences yi e B* and ki G N with 

i'(x)=sup 2 k n yt (x) = ^22 k n yt (x), 

i 

with the property that if i < j and l yi (x) = l y . (x) = 1 then ki < kj. The equality = follows 
easily from the fact that for any finite sequence n\ < n-i < . . . , £^ ■ 2 nj ^ 2 max.,- 2 n i . Since 

ptf < 1, we have ^2 i 2 ki fi(yi) < 1. Since the function i — > 2 ki p(yi) is computable, this 
implies 2 ki p(yi) < m(i), 2 ki < m(i)/m(yj). Thus, 

m(z) 



t(x) < sup l Vi (x) 



For y € N* we certainly have < mi y=yi H{i), which implies sup y=?/ . m(i) ^ m(y). It 
follows that 

* \ m (y) m(x^ n ) 

tlx) < sup l«(x) — -r^- = sup — . ■? r . 

Taking logarithms, we obtain the < part of the theorem. □ 
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Proof of Theorem 5. The proof of part > of the inequality follows directly from Proposi- 
tion 4.12, just as in the proof of Theorem 4. 

The proof of < is also similar to the proof of that theorem. The only part that needs to 
be reproved is the statement that for every lower semicomputable function / over X, there 
are computable sequences j/j £ B* and qi e Q with f(x) = supj qil yi (x). This follows now, 
since according to Proposition 2.10, the cells T y form a basis of the space X. □ 

5. Neutral measure 

Let tn(x) be our universal uniform randomness test. We call a measure M neutral if 
^m(x) ^ 1 for all x. If M is neutral then no experimental outcome x could refute the 
theory (hypothesis, model) that M is the underlying measure to our experiments. It can be 
used as "apriori probability", in a Bayesian approach to statistics. Levin's theorem says the 
following: 

Theorem 6. If the space X is compact then there is a neutral measure over X. 

The proof relies on a nontrivial combinatorial fact, Sperner's Lemma, which also un- 
derlies the proof of the Brouwer fixpoint theorem. Here is a version of Sperner's Lemma, 
spelled out in continuous form: 

Proposition 5.1 (see for example [20]). Let pi,...,pf. be points of some finite-dimensional 
space R". Suppose that there are closed sets Fi, . . . , with the property that for every subset 
1 < i\ < ■ ■ ■ < ij ^ k of the indices, the simplex S(pi 1 , . . . ,py) spanned by p^,. . . ,pi j is 
covered by the union F^ U • • • U F iy Then the intersection f] { Fi of all these sets is not empty. 

The following lemma will also be needed. 

Lemma 5.2. For every closed set A c X and measure fi, if n(A) = 1 then there is a point 
x e A with t At (x) sc: l. 

Proof. This follows easily from fit^ = fi x l^aO^x) ^ 1. □ 

Proof of Theorem 6. For every point ieX, let F x be the set of measures for which t M (x) ^ 
1. If we show that for every finite set of points xi, . . . , x^, we have 

4n-n^^, (5.1) 

then we will be done. Indeed, according to Proposition B.18, the compactness of X implies 
the compactness of the space .M(X) of measures. Therefore if every finite subset of the 
family { F x : x 6 X } of closed sets has a nonempty intersection, then the whole family has 
a nonempty intersection: this intersection consists of the neutral measures. 

To show (5.1), let S(x\, . . . , Xk) be the set of probability measures concentrated on 
x\,... ,Xk- Lemma 5.2 implies that each such measure belongs to one of the sets F Xi . 
Hence S(x±, . . . , Xk) C F Xl U • • • U F Xk , and the same holds for every subset of the indices 
{1, . . . , k}. Sperner's Lemma 5.1 implies F Xl n • • • D F Xk / 0. □ 

When the space is not compact, there are generally no neutral probability measures, as 
shown by the following example. 

Proposition 5.3. Over the discrete space X = N of natural numbers, there is no neutral 
measure. 
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Proof. It is sufficient to construct a randomness test t^(x) with the property that for every 
measure \i, we have sup x t tl (x) = oo. Let 

t„(x) = sup{ teN:^ n(y) > 1 - 2" k }. (5.2) 

By its construction, this is a lower semicomputable function with sup^t^x) = oo. It is a 
test if Y, x v{ x )tii{x) < 1. We have 

Em^o^E E ^)<E 2 ~ fc ^ L 

□ 

Using a similar construction over the space N w of infinite sequences of natural numbers, 
we could show that for every measure \i there is a sequence x with t M (x) = oo. 

Proposition 5.3 is a little misleading, since as a locally compact set, N can be compactified 
into N = N U {oo} (as in Part 1 of Example A.3). Theorem 6 implies that there is a neutral 
probability measure M over the compactified space N. Its restriction to N is, of course, not 
a probability measure, since it satisfies only Yl x <oo^( x ) ^ We cane d these functions 
semimeasures. 

Remark 5.4. 

1 . It is easy to see that Theorem 3 characterizing randomness in terms of complexity holds 
also for the space N. 

2. The topological space of semimeasures over N is not compact, and there is no neutral 
one among them. Its topology is not the same as what we get when we restrict the 
topology of probability measures over N to N. The difference is that over N, for example 
the set of measures { u. : /i(N) ^ 1/2 } is closed, since N (as the whole space) is a closed 
set. But over N, this set is not closed. 



Neutral measures are not too simple, even over N, as the following theorem shows. 

Theorem 7. There is no neutral measure over N that is upper semicomputable over N or lower 
semicomputable over N. 

Proof. Let us assume that v is a measure that is upper semicomputable over N. Then the set 

{ (x, r) : x € N, r € Q, v{x) < r } 

is recursively enumerable: let (xj,rj) be a particular enumeration. For each n, let i(n) be 
the first i with r{ < 2~ n , and let y n = x^ n y Then u(y n ) < 2~ n , and at the same time 

H{yn) < H(n). As mentioned, in Remark 5.4, Theorem 3 characterizing randomness in 
terms of complexity holds also for the space N. Thus, 

d v (y n ) = - log v{y n ) - H(y n \v)>n- H(n). 

Suppose now that v is lower semicomputable over N. The proof for this case is longer. 
We know that v is the monotonic limit of a recursive sequence i i— > Vi(x) of recursive 
semimeasures with rational values i^(x). For every k = 0, . . . , 2 n — 2, let 

V n ,k = {/*£ M(N) : k ■ 2~ n < M ({0, . . . , 2 n - 1}) < (k + 2) ■ 2~ n }, 
J = { (n, k) : k ■ 2~ n < zv({0, . . . , 2™ - 1}) }. 
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The set J is recursively enumerable. Let us define the functions j : J — > N and x : J — > 
{0, . . . , 2 n - 1} as follows: jfn, fc) is the smallest i with z/;({0, . . . , 2 n - 1}) > k ■ 2~ n , and 

x n>fc = min{y < 2 n : Vj{n,k){y) < 2" n+1 }. 

Let us define the function / M (x, n, fc) as follows. We set f^(x, n, k) = 2 n ~ 2 if the following 
conditions hold: 

(a) jx G F nifc ; 

(b) n(x) < 2" n+2 ; 

(c) (n, k) G J and x = x n fc. 

Otherwise, /^(x, n, fc) = 0. Clearly, the function (//, x, n, fc) i— > /^(x, n, fc) is lower semicom- 
putable. Condition (b) implies 

Y, »(y)U(y> ™> k ) < ^K^/m^* ™> fc ) < 2_n+2 • 2n " 2 = !■ ( 5 - 3 ) 

2/ 

Let us show that i/ G T4,fc implies 

U{x njk ,n,k) = 2 n - 2 . (5.4) 

Consider x = x ni &. Conditions (a) and (c) are satisfied by definition. Let us show that 
condition (b) is also satisfied. Let j = j(n, k). By definition, we have Vj(x) < 2~ n+1 . Since 
by definition Uj G V n> k and Uj < u G F n ,fc, we have 

i/(x) ^(x) + 2~ n+1 < 2" n+1 + 2~ n+1 = 2~ n+2 . 

Since all three conditions (a), (b) and (c) are satisfied, we have shown (5.4). Now we 
define 

n^2 y ' ' k 

Let us prove that g^{x) is a uniform test. It is lower semicomputable by definition, so we 
only need to prove / u(x)/ M (x) ^ 1. For this, let = {k : n G V n ,k}- Clearly by 
definition, \I n ,^\ ^ 2. We have, using this last fact and the test property (5.3): 



YVx)g^(x) = V r — - V V/i(x)/ At (x,ra,fc) < V 1 • 2 < 1. 

x n>2 v ; feeing a; n^2 v ' 

Thus, 5^ (x) is a uniform test. If v G V^^ then we have 



1 



in-2 



n(n + 1) > ' ' _|_ 
Hence v is not neutral. □ 

Remark 5.5. In [12] and [13], Levin imposed extra conditions on tests which allow to find a 
lower semicomputable neutral semimeasure. A typical (doubtless reasonable) consequence 
of these conditions would be that if outcome x is random with respect to measures /x and u 
then it is also random with respect to (fi + u)/2. 

Remark 5.6. The universal lower semicomputable semimeasure m(x) has a certain property 
similar to neutrality. According to Theorem 3, for every computable measure /i we have 
d M (x) — — log/i(x) — H(x) (where the constant in = depends on //). So, for computable 
measures, the expression 

d„(x) = -\og»(x)-H(x) (5.5) 
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can serve as a reasonable deficiency of randomness. (We will also use the test t = 2 d .) If we 
substitute m for [i in d^(x), we get 0. This substitution is not justified, of course. The fact 
that m is not a probability measure can be helped, at least over N, using compactification 
as above, and extending the notion of randomness tests. But the test d M can replace d M 
only for computable fx, while m is not computable. Anyway, this is the sense in which all 
outcomes might be considered random with respect to m, and the heuristic sense in which 
m may still be considered "neutral". <0 

Remark 5.7. Solomonoff proposed the use of a universal lower semicomputable semimea- 
sure (actually, a closely related structure) for inductive inference in [18]. He proved in [19] 
that sequences emitted by any computable probability distribution can be predicted well by 
his scheme. It may be interesting to see whether the same prediction scheme has stronger 
properties when used with the truly neutral measure M of the present paper. <0 

6. Relative entropy 

Some properties of description complexity make it a good expression of the idea of indi- 
vidual information content. 

6.1. Entropy. The entropy of a discrete probability distribution n is defined as 

X 

To generalize entropy to continuous distributions the relative entropy is defined as follows. 
Let n,u be two measures, where fi is taken (typically, but not always), to be a probabil- 
ity measure, and v another measure, that can also be a probability measure but is most 
frequently not. We define the relative entropy H u (fi) as follows. If fi is not absolutely con- 
tinuous with respect to v then H u (fi) = — oo. Otherwise, writing 

dp, = pjdx) = _ 
dv u(dx) 

for the (Radon-Nikodym) derivative (density) of p with respect to v, we define 

HM = - [ loAp = -^\og^\ = -!/*/(*) log /(a). 
J dv u{dx) 

Thus, H(p) = H#(p) is a special case. 

Example 6.1. Let f(x) be a probability density function for the distribution p over the real 
line, and let A be the Lebesgue measure there. Then 

H x (p) = - J f (x) log f(x)dx. 



In information theory and statistics, when both p and v are probability measures, then 
—TL v {pL) is also denoted D(p \\ v), and called (after Kullback) the information divergence 
of the two measures. It is frequently used in the role of a distance between /i and v. It is 
not symmetric, but can be shown to obey the triangle inequality, and to be nonnegative. Let 
us prove the latter property: in our terms, it says that relative entropy is nonpositive when 
both /i and v are probability measures. 
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Proposition 6.2. Over a space X, we have 

W^X-jipOlog^S- (6-1) 
In particular, if p(X) ^ v(X) then H v {p) < 0. 

Proof. The inequality —a In a < —a In 6 + 6 — a expresses the concavity of the logarithm 
function. Substituting a = f(x) and b = p(X)/v(X) and integrating by v: 

Qn2)HM = -v x f{x)\nf{x) < -^(X)ln^j + ^|L(X) - = - M (X)ln^|||, 

giving (6.1). □ 

The following theorem generalizes an earlier known theorem stating that over a discrete 
space, for a computable measure, entropy is within an additive constant the same as "aver- 
age complexity": H(p) = p x H(x). 

Theorem 8. Let pbe a probability measure. Then we have 

U v {p) < fJ, x H u (x | p). (6.2) 
If X is a discrete space then the following estimate also holds: 

H v (ji) > p x H v {x | p). (6.3) 

Proof. Let 8 be the measure with density t u (x \ p) with respect to u: t u (x \ p) = ^^y- Then 
S(X) ^ 1. It is easy to see from the maximality property of t u (x \ p) that t v (x \ p) > 0, 

therefore according to Proposition B.9, we have = {^j^^j ■ Using Proposition B.9 
and 6.2: 

vyax) 

-p x H u (x \p)= p x log -j- ^ = log ■ 



v{dx) 5(dx 
H u (p) - p x H u (x | p) = —p x log ^ < -/x(X) log < o. 

This proves (6.2). 



Over a discrete space X, the function (x, p, v) i— > = is computable, there- 

/i(cix) 
i/(dx) 

"i> n* FT fi» I /i^ 



fore by the maximality property of | /x) we have ^fej < t v (x | /i), hence TL u {p) = 

-p X \0g^r>p X H v { X \p). □ 



6.2. Addition theorem. The most important information-theoretical property of descrip- 
tion complexity is the following theorem (see for example [14]): 

Proposition 6.3 (Addition Theorem). We have H(x,y) = H(x) + H(y \ x,H(x)). 

Mutual information is defined as I(x : y) = H(x) + H(y) — H(x,y). By the Addition 
theorem, we have I(x : y) = H(y) — H(y \ x, H(x)) = H(x) — H(x \ y, H(y)). The two latter 
expressions show that in some sense, I(x : y) is the information held in x about y as well as 
the information held in y about x. (The terms H(x), H(y) in the conditions are logarithmic- 
sized corrections to this idea.) Using (5.5), it is interesting to view mutual information 
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I(x : y) as a deficiency of randomness of the pair (x, y) in terms of the expression d M , with 
respect to m x m: 

I(x : y) = H(x) + H(y) - H(x, y) = d mxm (x, y). 

Taking m as a kind of "neutral" probability, even if it is not quite such, allows us to view 
I(x : y) as a "deficiency of independence". Is it also true that I(x : y) = d mxm (x)? This 
would allow us to deduce, as Levin did, "information conservation" laws from randomness 
conservation laws. 1 

Expression d mxm (rr) must be understood again in the sense of compactification, as in 
Section 5. There seem to be two reasonable ways to compactify the space N x N: we either 
compactify it directly, by adding a symbol oo, or we form the product N x N. With either of 
them, preserving Theorem 3, we would have to check whether H(x, y | m x m) = H(x, y). 
But, knowing the function m(x) x m(y) we know the function x i— * m(x) = m(x) x m(0), 
hence also the function (x, y) i— > m(a;, y) = m((x, y)), where (x, y) is any fixed computable 
pairing function. Using this knowledge, it is possible to develop an argument similar to the 
proof of Theorem 7, showing that H(x, y | m x m) = H(x, y) does not hold. 

Question 1. 7s there a neutral measure M with the property I(x : y) = d.MxM(x,y)? Is this 
true maybe for all neutral measures M? If not, how far apart are the expressions (1mxm(x, y) 
and I(x : y) from each other? 

The Addition Theorem (Proposition 6.3) can be generalized to the algorithmic entropy 
Hfj,(x) introduced in (4.1) (a somewhat similar generalization appeared in [23]). The gen- 
eralization, defining H^ v = H^ lXU , is 

H^ u (x, y) = Hfj,(x | v) + H u (y \ x, H^{x \u), fx). (6.4) 

Before proving the general addition theorem, we establish a few useful facts. 

Proposition 6.4. We have 

H^x | v) < - \ogv y 2- H ^ x ' y \ 

Proof. The function f(x,fi,u) that is the right-hand side, is upper semicomputable by def- 
inition, and obeys ^ x 2~^ x ' fl ' u ^ ^ 1. Therefore the inequality follows from the minimum 
property of (x). □ 

Let us generalize the minimum property of H^x). 

Proposition 6.5. Let (x,y,u) >— > f u (x,y) be a nonnegative lower semicomputable function 
with F v (x) = log is y f u (x, y). Then for all x with F v (x) > — oo we have 

H v {y | x,[F v (x)\) < -logf u (x,y) + F u (x). 

Proof. Let us construct a lower semicomputable function (x,y,m,u) i— > g u (x,y,m) for in- 
tegers m with the property that v y g u (x,y,m) ^ 2~ m , and for all x with F u (x) ^ — m 
we have g v {x 1 y 1 m) = f u (x,y). Such a g can be constructed by watching the approx- 
imation of / grow and cutting it off as soon as it would give F v (x) > —m. Now 
(x, y, m, v) i — > 2 m g u (x, y, m) is a uniform conditional test of y and hence it is < 2~ H "^ x ' m > . 
To finish the proof, substitute — L-^(^)J for fn an d rearrange. □ 

"^We cannot use the test t M for this, since — as it can be shown easily-it does not obey randomness 
conservation. 
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Let zeN, then the inequality 

H^x) < H(z) + H^x \ z) (6.5) 

will be a simple consequence of the general addition theorem. The following lemma, 
needed in the proof of the theorem, generalizes this inequality somewhat: 

Lemma 6.6. For a computable function (y, z) i— > f(y, z) over N, we have 

H lt (x\y)<H(z)+H li (x\f(y,z)). 

Proof. The function 

(x, y, fj) » g,(x, y)=J2 2- H *Wto,*))-H{z) 

z 

is lower semicomputable, and » x 9n(x,y) < E z 2~ H(z) < 1. Hence g^{x,y) < 2~ a ^ x \v\ The 
left-hand side is a sum, hence the inequality holds for each element of the sum: just what 
we had to prove. □ 

As mentioned above, the theory generalizes to measures that are not probability mea- 
sures. Taking f^{x,y) = 1 in Proposition 6.5 gives the inequality 

H^x | Itog/ipOJ) <log/i(*), 
with a physical meaning when /x is the phase space measure. Using (6.5), this implies 

H^x) < log/x(X) +H{\\ogn{X)\). (6.6) 
The following simple monotonicity property will be needed: 
Lemma 6.7. For i < j we have 

i + H^(x | i) <j + H^(x 
Proof. From Lemma 6.6, with f(i, n) = % + n we have 

H^x | i) - H^x | j) < H(j -i)<j-i. 

□ 

Theorem 9 (General addition). The following inequality holds: 

H^ v {x,y) = H^(x | u) + H v (y \ x, H^(x \ v), u). 

Proof. To prove the inequality <, let us define 

Gu,v(x,y,m) = min i + H u (y \ x,i,fi). 

Function G IJL)V (x, y, m) is upper semicomputable and decreasing in m. Therefore 

G^ v {x,y) = G^ v {x,y,H^{x \ v)) 

is also upper semicomputable since it is obtained by substituting an upper semicomputable 
function for m in G^ u (x, y, m). Lemma 6.7 implies 

G^ u (x, y,m) =m + H u (y \ x, m, /i), 

Gfj,,v(x, y) = H^x | v) + H v {y \ x, H^(x \ v),y). 
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Now, we have 

l/ y2~m~H v (y\x,m.,fj,) ^ 2~ m , 

Therefore \i x v v 2~ G < 1, implying H fl ^ u (x,y) < G^ jU (x,y) by the minimality property of 
H^ jU (x, y). This proves the < half of our theorem. 
To prove the inequality >, let 

U(x,y,»)=2- H ^>y\ 
F v (x,n) =logu y f u (x,y,n). 

According to Proposition 6.5, 

H v {y | x, \_F\,n) < - log f v (x,y, n) + F u (x, (j,), 
H^ u (x,y) > -F + H u (y \ x, [—F],fx). 

Proposition 6.4 implies —F 1/ (x,^l) > H^x \ v). The monotonity lemma 6.7 implies from 
here the > half of the theorem. □ 

6.3. Some special cases of the addition theorem; information. The function H^(-) be- 
haves quite differently for different kinds of measures fi. Recall the following property of 
complexity: 

H(f(x) | y) < H(x | g{y)) < H(x). (6.7) 
for any computable functions /, g This implies 

H(y)<H(x,y). 
In contrast, if ^ is a probability measure then 

H u (y) > H^ v (x,y). 

This comes from the fact that 2~ Hv ^ is a test for fixi/. 

Let us explore some of the consequences and meanings of the additivity property. As 
noted in (4.2), the subscript /i can always be added to the condition: H^( x ) — H^x | fi). 
Similarly, we have 

H^ l)U (x, y) := H MiXU (x, y) = H MiXU (x, y \ (j, X v) = H^ xv (x, y \ H,v) =: H^ v (x, y \ fi, u), 

where only before-last inequality requires new (easy) consideration. 

Let us assume that X = Y = £*, the discrete space of all strings. With general fi, u such 
that fi(x),u(x) / for all x, using (4.6), the addition theorem specializes to the ordinary 
addition theorem, conditioned on fi, v: 

H(x, y | fj,,u) = H(x | /i, v) + H(y \ x, H(x \ fi, u), /x, v). 

In particular, whenever /i, v are computable, this is just the regular addition theorem. 

Just as above, we defined mutual information as I(x : y) = H(x) + H(y) — H(x, y), the 
new addition theorem suggests a more general definition 

i^A x : v) = H n( x I v ) + H v(y I m) - H nA x iv)- 

In the discrete case X = Y = S* with everywhere positive fi(x),b'(x), this simplifies to 

i^A x : v) = H ( x I Mi u ) + H (y I a*i v ) - H ( x i v )> 
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which is = I(x : y) in case of computable /i, v. How different can it be for non-computable 

In the general case, even for computable fi, v, it seems worth finding out how much this 
expression depends on the choice of (i, v. Can one arrive at a general, natural definition of 
mutual information along this path? 

7. Conclusion 

When uniform randomness tests are defined in as general a form as they were here, 
the theory of information conservation does not fit nicely into the theory of randomness 
conservation as it did with [12] and [13]. Still, it is worth laying the theory onto broad 
foundations that, we hope, can serve as a basis for further development. 

Appendix A. Topological spaces 

Given two sets X, Y, a partial function f from X to Y, defined on a subset of Y, will be 
denoted as 

/ :C X - Y. 

A.l. Topology. A topology on a set X is defined by a class t of its subsets called open 
sets. It is required that the empty set and X are open, and that arbitrary union and finite 
intersection of open sets is open. The pair (X, r) is called a topological space. A topology 
t' on X is called larger, or finer than r if r' D r. A set is called closed if its complement is 
open. A set B is called the neighborhood of a set A if B contains an open set that contains 
A. We denote by A, A the closure (the intersection of all closed sets containing A) and the 
interior of A (the union of all open sets in A) respectively. Let 

dA = A \ A° 

denote the boundary of set A. A base is a subset (3 of r such that every open set is the union 
of some elements of (3. A neighborhood of a point is a base element containing it. A base 
of neighborhoods of a point x is a set N of neighborhoods of x with the property that each 
neighborhood of x contains an element of N. A subbase is a subset a of r such that every 
open set is the union of finite intersections from a. 

Examples A.l. 

1. Let X be a set, and let (3 be the set of all points of X. The topology with base (3 is the 
discrete topology of the set X. In this topology, every subset of X is open (and closed). 

2. Let X be the real line M., and let /% be the set of all open intervals (a; b). The topology 
tr obtained from this base is the usual topology of the real line. When we refer to R as a 
topological space without qualification, this is the topology we will always have in mind. 

3. Let X = R = MU {— oo, oo}, and let (3^ consist of all open intervals (a; b) and in addition 
of all intervals of the forms [— oo; a) and (a; oo]. It is clear how the space M + is defined. 

4. Let X be the real line E. Let 0^ be the set of all open intervals (— oo; b). The topology 
with base 0^ is also a topology of the real line, different from the usual one. Similarly, 
let (3^ be the set of all open intervals (b; oo). 

5. On the space S w , let t c = { AT^ : A C S* } be called the topology of the Cantor space 
(over E) . 
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A set is called a G$ set if it is a countable intersection of open sets, and it is an F a set if it 
is a countable union of closed sets. 

For two topologies n, t 2 over the same set X, we define the topology t\ V r 2 = n n r 2 , 
and n A r 2 as the smallest topology containing n U r 2 . In the example topologies of the real 
numbers above, we have tr = At|". 

We will always require the topology to have at least the To property: every point is 
determined by the class of open sets containing it. This is the weakest one of a number of 
other possible separation properties: both topologies of the real line in the example above 
have it. A stronger such property would be the T 2 property: a space is called a Hausdorff 
space, or T 2 space, if for every pair of different points x, y there is a pair of disjoint open 
sets A, B with x e A, y G B. The real line with topology in Example A. 1.4 above is not 
a Hausdorff space. A space is Hausdorff if and only if every open set is the union of closed 
neighborhoods. 

Given two topological spaces (Xj,Tj) (i = 1,2), a function / :C X\ — > X 2 is called 
continuous if for every open set G C X 2 its inverse image f^ 1 (G) is also open. If the topolo- 
gies ri, r 2 are not clear from the context then we will call the function (n, r 2 ) -continuous. 
Clearly, the property remains the same if we require it only for all elements G of a subbase 
of X 2 . If there are two continuous functions between X and Y that are inverses of each 
other then the two spaces are called homeomorphic. We say that / is continuous at point x 
if for every neighborhood V of f(x) there is a neighborhood U of x with f(U) C V. Clearly, 
/ is continuous if and only if it is continuous in each point. 

A subspace of a topological space (X, r) is defined by a subset Y C X, and the topology 
Ty = {G DY : G £ t}, called the induced topology on Y. This is the smallest topology on 
Y making the identity mapping x i— > x continuous. A partial function / :C X — > Z with 
dom(/) = Y is continuous iff / : Y — > Z is continuous. 

For two topological spaces {X^rA (i = 1,2), we define the product topology on their 
product X x Y: this is the topology defined by the subbase consisting of all sets G± x X 2 
and all sets X\ x G 2 with Gj £ 73. The product topology is the smallest topology making the 
projection functions (a;, y) 1— > x, (x, y) 1— > y continuous. Given topological spaces X, Y, Z we 
call a two-argument function / : X x Y 1— > Z continuous if it is continuous as a function 
from X x Y to Z. The product topology is defined similarly for over the product Yl ieI Xi 
of an arbitrary number of spaces, indexed by some index set /. We say that a function is 
(ri, . . . , r n , fj) -continuous if it is (t± x ■ ■ ■ x r n , n) -continuous. 

Examples A. 2. 

1 . The space RxM with the product topology has the usual topology of the Euclidean plane. 

2. Let X be a set with the discrete topology, and X^ the set of infinite sequences with 
elements from X, with the product topology. A base of this topology is provided by all 
sets of the form uX^ where u e X*. The elements of this base are closed as well as open. 
When X = {0, 1} then this topology is the usual topology of infinite binary sequences. 

A real function / : X\ — > R is called continuous if it is (n, ^-continuous. It is called 
lower semicontinuous if it is (n, Tj^)-continuous. The definition of upper semicontinuity is 
similar. Clearly, / is continuous if and only if it is both lower and upper semicontinuous. 
The requirement of lower semicontinuity of / is that for each r € R, the set { x : f(x) > r } 
is open. This can be seen to be equivalent to the requirement that the single set { (x, r) : 
f(x) > r } is open. It is easy to see that the supremum of any set of lower semicontinuous 
functions is lower semicontinuous. 
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Let (X, t) be a topological space, and B a subset of X. An open cover of B is a family of 
open sets whose union contains B. A subset K of X is said to be compact if every open cover 
of K has a finite subcover. Compact sets have many important properties: for example, a 
continuous function over a compact set is bounded. 

Example A. 3. 

1. Every finite discrete space is compact. An infinite discrete space X = (X, r) is not 
compact, but it can be turned into a compact space X by adding a new element called 
oo: let X = X U {oo}, and r = r U {X \ A : A c X closed }. More generally, this 
simple operation can be performed with every space that is locally compact, that each of 
its points has a compact neighborhood. 

2. In a finite-dimensional Euclidean space, every bounded closed set is compact. 

3. It is known that if (Xi)j g / is a family of compact spaces then their direct product is also 
compact. 



A subset K of X is said to be sequentially compact if every sequence in K has a convergent 
subsequence with limit in K. The space is locally compact if every point has a compact 
neighborhood. 

A.2. Metric spaces. In our examples for metric spaces, and later in our treatment of the 
space of probability measures, we refer to [2]. A metric space is given by a set X and a 
distance function d : X xX ^ M.+ satisfying the triangle inequality d(x, z) ^ d(x, y) + d(y, z) 
and also property that d(x, y) = implies x = y. For r 6 K + , the sets 

B(x,r) = {y : d(x,y) < r}, B(x,r) = {y : d(x,y) ^ r} 

are called the open and closed balls with radius r and center x. A metric space is also 
a topological space, with the base that is the set of all open balls. Over this space, the 
distance function d(x, y) is obviously continuous. Each metric space is a Hausdorff space; 
moreover, it has the following stronger property. For every pair of different points x, y there 
is a continuous function / : X — > E with f(x) ^ f(y)- (To see this, take f(z) = d(x,z).~) 
This is called the T3 property. A metric space is bounded when d(x, y) has an upper bound 
on X. A topological space is called metrizable if its topology can be derived from some 
metric space. 

Notation. For an arbitrary set A and point x let 

d{x,A) = inf d{x,y), 

A £ = {x:d(x,A) <e}. (A.l) 

Examples A A. 

1. The real line with the distance d(x,y) = \x — y\ is a metric space. The topology of this 
space is the usual topology tr of the real line. 

2. The space 1 x R with the Euclidean distance gives the same topology as the product 
topology ofixM. 

3. An arbitrary set X with the distance d(x, y) = 1 for all pairs x, y of different elements, is 
a metric space that induces the discrete topology on X. 
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4. Let X be a bounded metric space, and let Y = X w be the set of infinite sequences 
x = (x\, X2, ■ ■ ■ ) with distance function d w (x, y) = J2i 2~ l d(xi, Vi)- The topology of this 
space is the same as the product topology defined in Example A.2.2. 

5. Let X be a metric space, and let Y = X w be the set of infinite bounded sequences 
x = (x±, X2, ■ ■ ■ ) with distance function d(x, y) = supj d(xi,yi). 

6. Let X be a metric space, and let C(X) be the set of bounded continuous functions over 
X with distance function d'(f,g) = swp x d(f(x),g(x)). A special case is C[0; 1] where 
the interval [0; 1] of real numbers has the usual metric. 

7. Let I2 be the set of infinite sequences x = (x\, X2, ■ ■ ■ ) of real numbers with the property 
that £V x\ < 00. The metric is given by the distance d(x, y) = (£V \xi — y^ 2 ) 1 / 2 . 



A topological space has the first countability property if each point has a countable base of 
neighborhoods. Every metric space has the first countability property since we can restrict 
ourselves to balls with rational radius. Given a topological space (X, r) and a sequence x = 
(x\, X2, ■ ■ ■ ) of elements of X, we say that x converges to a point y if for every neighborhood 
G of y there is a k such that for all m > k we have x m e G. We will write y = lim n ^oo x n . 
It is easy to show that if spaces (Xj, Tj) (i = 1,2) have the first countability property then 
a function / : X — > Y is continuous if and only if for every convergent sequence (x n ) we 
have /(lim n x n ) = lim n f(x n ). A topological space has the second countability property if the 
whole space has a countable base. For example, the space E has the second countability 
property for all three topologies tr, t^, t|T. Indeed, we also get a base if instead of taking 
all intervals, we only take intervals with rational endpoints. On the other hand, the metric 
space of Example A.4.5 does not have the second countability property. In a topological 
space (X, t), a set B of points is called dense at a point x if it intersects every neighborhood 
of x. It is called everywhere dense, or dense, if it is dense at every point. A metric space is 
called separable if it has a countable everywhere dense subset. This property holds if and 
only if the space as a topological space has the second countability property. 

Example A.5. In Example A.4.6, for X = [0; 1], we can choose as our everywhere dense set 
the set of all polynomials with rational coefficients, or alternatively, the set of all piecewise 
linear functions whose graph has finitely many nodes at rational points. (} 

Let X be a metric space, and let C{X) be the set of bounded continuous functions over 
X with distance function d'(f,g) = swp x d(f(x),g(x)). A special case is C[0; 1] where the 
interval [0; 1] of real numbers has the usual metric. 

Let (X, d) be a metric space, and a = (ai, ai, . . . ) an infinite sequence. A metric space is 
called complete if every Cauchy sequence in it has a limit. It is well-known that every metric 
space can be embedded (as an everywhere dense subspace) into a complete space. 

It is easy to see that in a metric space, every closed set is a G$ set (and every open set is 
an F a set) . 

Example A.6. Consider the set D[0; 1] of functions over [0; 1] that are right continuous and 
have left limits everywhere. The book [2] introduces two different metrics for them: the 
Skorohod metric d and the do metric. In both metrics, two functions are close if a slight 
monotonic continuous deformation of the coordinate makes them uniformly close. But in 
the c?o metric, the slope of the deformation must be close to 1 . It is shown that the two 
metrics give rise to the same topology; however, the space with metric d is not complete, 
and the space with metric do is. <C> 
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Let (X, d) be a metric space. It can be shown that a subset K of X is compact if and only 
if it is sequentially compact. Also, K is compact if and only if it is closed and for every e, 
there is a finite set of e-balls (balls of radius e) covering it. 

We will develop the theory of randomness over separable complete metric spaces. This 
is a wide class of spaces encompassing most spaces of practical interest. The theory would 
be simpler if we restricted it to compact or locally compact spaces; however, some impor- 
tant spaces like C[0; 1] (the set of continuouos functions over the interval [0; 1], with the 
maximum difference as their distance) are not locally compact. 

Given a function / : X — > Y between metric spaces and (3 > 0, let Lip^(X, Y) denote the 
set of functions (called the Lipschitz functions, or simply Lipschitz functions) satisfying 

dY(f(x)J(y))^(3d x (x,y). (A.2) 

All these functions are uniformly continuous. Let Lip(X) = Lip(X,R) = |L Lip^ be the set 
of real Lipschitz functions over X. 

Appendix B. Measures 

For a survey of measure theory, see for example [16]. 

B.l. Set algebras. A (Boolean set-) algebra is a set of subsets of some set X closed under 
intersection and complement (and then, of course, under union) . It is a o-algebra if it is also 
closed under countable intersection (and then, of course, under countable union) . A semi- 
algebra is a set C of subsets of some set X closed under intersection, with the property that 
the complement of every element of C is the disjoint union of a finite number of elements 
of C. If C is a semialgebra then the set of finite unions of elements of C is an algebra. 

Examples B.l. 

1. The set L\ of left-closed intervals of the line (including intervals of the form (— oo; a)) is 
a semialgebra. 

2. The set £2 of all intervals of the line (which can be open, closed, left-closed or right- 
closed), is a semialgebra. 

3. In the set {0, 1} U of infinite 0-1 -sequences, the set £3 of all subsets of the form u{0, 1} U 
with u G {0, 1}*, is a semialgebra. 

4. The cr-algebra B generated by £\, is the same as the one generated by £2, and is also the 
same as the one generated by the set of all open sets: it is called the family of Borel sets 
of the line. The Borel sets of the extended real line E are defined similarly. 

5. Given u-algebras A, B in sets X, Y, the product cr-algebra A x B in the space X x Y is 
the one generated by all elements A x Y and X x B for A e A and B € B. 



B.2. Measures. A measurable space is a pair (X, S) where S is a cr-algebra of sets of X. 
A measure on a measurable space (X, S) is a function \i : B — > M. + that is a-additive: 
this means that for every countable family A±,A2,... of disjoint elements of S we have 
M(Ui Ai) = Yli ^(Ai)- A measure /i is a-finite if the whole space is the union of a countable 
set of subsets whose measure is finite. It is finite if n(X) < 00. It is a probability measure if 
ti{X) = 1. 

It is important to understand how a measure can be defined in practice. Algebras are 
generally simpler to grasp constructively than cr-algebras; semialgebras are yet simpler. 
Suppose that /i is defined over a semialgebra C and is additive. Then it can always be 
uniquely extended to an additive function over the algebra generated by C The following 
is an important theorem of measure theory. 
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Proposition B.2. Suppose that a nonnegative set function defined over a semialgebra £ is 
a-additive. Then it can be extended uniquely to the a-algebra generated by £. 

Examples B.3. 

1. Let x be point and let p(A) = 1 if x G A and otherwise. In this case, we say that p is 
concentrated on the point x. 

2. Consider the the line R, and the algebra L\ defined in Example B.l.l. Let / : R — > R be 
a monotonic real function. We define a set function over L\ as follows. Let [c^; hi), (i = 
1, . . . , n) be a set of disjoint left-closed intervals. Then //(UJ a «; = /(M ~~ /( a «)- 
It is easy to see that p is additive. It is cr-additive if and only if / is left-continuous. 

3. Let B = {0, 1}, and consider the set B w of infinite 0-1-sequences, and the semialgebra 
£3 of Example B.1.3. Let p : B* —> R + be a function. Let us write p(uB w ) = p(u) 
for all u G B*. Then it can be shown that the following conditions are equivalent: p is 
a-additive over £3; it is additive over £3; the equation p(u) = p(u0) + p(ul) holds for 
all u G B*. 

4. The nonnegative linear combination of any finite number of measures is also a measure. 
In this way, it is easy to construct arbitrary measures concentrated on a finite number of 
points. 

5. Given two measure spaces (X,A,p) and (Y,B, v) it is possible to define the product 
measure p x v over the measureable space ( X x Y, A x B) . The definition is required to 
satisfy p x u(A x B) = p(A) x u(B), and is determined uniquely by this condition. If v 
is a probability measure then, of course, p(A) = p x u(A x Y). 



Remark B.4. Example B.3.3 shows a particularly attractive way to define measures. Keep 
splitting the values p(u) in an arbitrary way into p(u0) and p(ul), and the resulting values 
on the semialgebra define a measure. Example B.3.2 is less attractive, since in the process 
of defining p on all intervals and only keeping track of finite additivity, we may end up 
with a monotonic function that is not left continuous, and thus with a measure that is not 
cr-additive. In the subsection on probability measures over a metric space, we will find that 
even on the real line, there is a way to define measures in a step-by-step manner, and only 
checking for consistency along the way. 

A probability space is a triple (X, S, P) where (X, S) is a measurable space and P is a 
probability measure over it. 

Let (Xi,Si) (i = 1,2) be measurable spaces, and let / : X — > Y be a mapping. Then / 
is measurable if and only if for each element B of S2, its inverse image is in S\. If 

pi is a measure over (Xi,Si) then p2 defined by p2(A) = pi(f~ 1 (A)) is a measure over X2 
called the measure induced by /. 

B.3. Integral. A measurable function / : X —> R is called a step function if its range is 
finite. The set of step functions is closed with respect to linear combinations and also with 
respect to the operations A, V. Such a set of functions is called a Riesz space. 

Given a step function which takes values X{ on sets A4, and a finite measure p, we define 



❖ 




This is a linear positive functional on the set of step functions. Moreover, it can be shown 
that it is continuous on monotonic sequences: if /j \ then pfi \ 0. The converse can 
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also be shown: Let \i be a linear positive functional on step functions that is continuous on 
monotonic sequences. Then the set function p(A) = h(1a) is a finite measure. 

Proposition B.5. Let £ be any Riesz space of functions with the property that 1 e £. Let fi 
be a positive linear functional on £ continuous on monotonic sequences, with fil = 1. The 
functional fi can be extended to the set £ + of monotonic limits of nonnegative elements of £, by 
continuity. In case when £ is the set of all step functions, the set £ + is the set of all nonnegative 
measurable functions. 

Let us fix a finite measure /i over a measurable space (X, S). A measurable function / 
is called integrable with respect to /i if < oo and p\f\~ < oo. In this case, we define 

A*/ = A t l/l + — /^l/l - The set of integrable functions is a Riesz space, and the positive linear 
functional /i on it is continuous with respect to monotonic sequences. The continuity over 
monotonic sequences also implies the following bounded convergence theorem. 

Proposition B.6. Suppose that functions f n are integrable and \f n \ < g for some integrable 
function g. Then f = lim n /„ is integrable and fif = lim n fif n . 

Two measurables functions /, g are called equivalent with respect to /i if fi(f — g) = 0. 
For two-dimensional integration, the following theorem holds. 

Proposition B.7. Suppose that function /(•, •) is integrable over the space (XxY,AxB,fix v). 
Then for ^-almost all x, the function f(x, •) is integrable over (Y, B, u), and the function x i— > 
v y 'f(x,y) is integrable over (X,A,p) with (/i x v)f = n x ^ y f. 

B.4. Density. Let fx, v be two measures over the same measurable space. We say that v is 
absolutely continuous with respect to fi, or that fi dominates v, if for each set A, n(A) = 
implies v(A) = 0. It can be proved that this condition is equivalent to the condition that 
there is a positive real number c with v ^ cfi. Every nonnegative integrable function / 
defines a new measure u via the formula v(A) = • 1^). This measure v is absolutely 
continuous with respect to fx. The Radon-Nikodym theorem says that the converse is also 
true. 

Proposition B.8 (Radon-Nikodym theorem). If v is dominated by /x then there is a nonneg- 
ative integrable function f such that v(A) = [i(f ■ 1a) for all measurable sets A. The function 
f is defined uniquely to within equivalence with respect to fi. 

The function / of the Radom-Nikodym Theorem above is called the density of v with 
respect to /x We will denote it by 

fj,(dx) dp, 

f( x ) = "7TT = 1~- 
ulax) av 

The following theorem is also standard. 
Proposition B.9. 

(a) Let n,u,rj be measures such that 77 is absolutely continuous with respect to and /i is 
absolutely continuous with respect to v. Then the "chain rule" holds: 

du dfi dv 

(b) If ^7gjY > for all x then p is also absolutely continuous with respect to v and ^|^y = 

/ v(dx) 
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Let /i, v be two measures, then both are dominated by some measure 77 (for example by 
77 = n + v). Let their densities with respect to 77 be / and g. Then we define the total 
variation distance of the two measures as 

D(fx,u) = V (\f-g\). 

It is independent of the dominating measure 77. 

Example B.10. Suppose that the space X can be partitioned into disjoint sets A, B such that 

v(A) = n{B) = 0. Then D(fi, v) = n(A) + v(B) = fj,(X) + v{X). 

B.5. Random transitions. Let (X, A), (Y,B) be two measureable spaces (defined in Sub- 
section B.2). We follow the definition given in [16]. Suppose that a family of probability 
measures A = { X x : x <E X } on B is given. We call it a probability kernel, (or Markov kernel, 
or conditional distribution) if the map x 1— > \ X B is measurable for each B e B. When X, Y 
are finite sets then A is a Markov transition matrix. The following theorem shows that A 
assigns a joint distribution over the space (X x Y, A x B) to each input distribution fi. 

Proposition B.ll. For each nonnegative A x B-measureable function f over X xY, 

1. the function y — ► f(x, y) is B-measurable for each fixed x; 

2. x — > Xxf(x,y) is A-measurable; 

3. the integral f — ► fi x X x f(x, y) defines a measure on Ax B. 

According to this proposition, given a probability kernel A, to each measure // over A 
corresponds a measure over Ax B. We will denote its marginal over B as 

A>. (B.2) 

For every measurable function g(y) over Y, we can define the measurable function f(x) = 
Kg = Kg(y)- we write 

/ = kg. (B.3) 
The operator A is linear, and monotone with Al = 1. By these definitions, we have 

//(As) = (A» 5 . (B.4) 

Example B.12. Let h : X — > Y be a measureable function, and let X x be the measure 
8 h r x \ concentrated on the point h(x). This operator, denoted A^ is, in fact, a deterministic 
transition, and we have Ahg = g o h. In this case, we will simplify the notation as follows: 

h*ti = At 



B.6. Probability measures over a metric space. We follow the exposition of [2]. When- 
ever we deal with probability measures on a metric space, we will assume that our metric 
space is complete and separable (Polish). Let X = (X,d) be a complete separable metric 
space. It gives rise to a measurable space, where the measurable sets are the Borel sets of 
X. It can be shown that, if A is a Borel set and fj, is a finite measure then there are sets 
F C A C G where F is an F a set, G is a G$ set, and /j.(F) = fi(G). Let B be a base of open 
sets closed under intersections. Then it can be shown that fj, is determined by its values on 
elements of B. The following proposition follows then essentially from Proposition B.2. 

Proposition B.13. Let B* be the set algebra generated by the above base B, and let /j be any 
a-additive set function on B* with fj,(X) = 1. Then fj, can be extended uniquely to a probability 
measure. 

We say that a set A is a continuity set of measure \x if n(dA) = 0: the boundary of A has 
measure 0. 
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B.6.1. Weak topology. Let 

M(X) 

be the set of probability measures on the metric space X. Let 

be a probability measure concentrated on point x. Let x n be a sequence of points converging 
to point x but with x n / x. We would like to say that 8 Xn converges to 5 X . But the total 
variation distance D(5 Xn ,5 x ) is 2 for all n. This suggests that the total variation distance is 
not generally the best way to compare probability measures over a metric space. We say 
that a sequence of probability measures [i n over a metric space (X, d) weakly converges to 
measure p, if for all bounded continuous real functions / over X we have [i n f — > [if. This 
topology of weak convergence (M.,t w ) can be defined using a number of different subbases. 
The one used in the original definition is the subbase consisting of all sets of the form 

A f,c = { V ■ M/ < c } 

for bounded continuous functions / and real numbers c. We also get a subbase (see for ex- 
ample [16]) if we restrict ourselves to the set Lip(X) of Lipschitz functions defined in (A.2). 
Another possible subbase giving rise to the same topology consists of all sets of the form 

B G>c = {[i:[x{G)>c} (B.5) 

for open sets G and real numbers c. Let us find some countable subbases. Since the space X 
is separable, there is a sequence Ui, l7 2 , . . . of open sets that forms a base. We can restrict 
the subbase of the space of measures to those sets Bq c where G is the union of a finite 
number of base elements Ui and c is rational. Thus, the space (M,t w ) itself has the second 
countability property. 

It is more convenient to define a countable subbase using bounded continuous functions 
/, since [i \— > [if is continuous on such functions, while [i i— > pXJ is typically not continuous 
when U is an open set. Let be the set of functions introduced before (2.1). Let 

be the set of functions / with the property that / is the minimum of a finite number of 
elements of JF . Note that each element / of T\ is bounded between and 1, and from its 
definition, we can compute a bound j3 such that / € Lip^. 

Proposition B.14. The following conditions are equivalent: 

1. [i n weakly converges to fi. 

2. [i n f -» [if for all f € T\. 

3. For every Borel set A, that is a continuity set of [i, we have [i n (A) — ► p,(A). 

4. For every closed set F, liminf n /i n (F) ^ n(F). 

5. For every open set G, limsup n [i n (G) < [i(G). 

As a subbase 

v M (B.6) 
for M(x), we choose the sets { [i : [if < r } and { [i : [if > r } for all / 6 T\ and r € Q. Let 
£ be the set of functions introduced in (2.1). It is a Riesz space as defined in Subsection B.3. 
A reasoning combining Propositions B.2 and B.5 gives the following. 

Proposition B.15. Suppose that a positive linear functional [i with [il = 1 is defined on £ 
that is continuous with respect to monotone convergence. Then fi can be extended uniquely to 
a probability measure over X with [if = f f (x) [i(dx) for all f € £. 
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B.6.2. Prokhorov distance. The definition of measures in the style of Proposition B.15 is not 
sufficiently constructive. Consider a gradual definition of the measure p, extending it to 
more and more elements of £, while keeping the positivity and linearity property. It can 
happen that the function fj, we end up with in the limit, is not continuous with respect to 
monotone convergence. Let us therefore metrize the space of measures: then an arbitrary 
measure can be defined as the limit of a Cauchy sequence of simple meaures. 

One metric that generates the topology of weak convergence is the Prokhorov distance 
p(fi, v): the infimum of all those e for which, for all Borel sets A we have (using the nota- 
tion (A.l)) 

A*(A) ^is(A £ ) + e. 

It can be shown that this is a distance and it generates the weak topology. The following 
result helps visualize this distance: 

Proposition B.16 (Coupling Theorem, see [21]). Let fi,u be two probability measures over 
a complete separable metric space X with p(p, v) ^ e. Then there is a probability measure P 
on the space X x X with marginals p and v such that for a pair of random variables (£,r/) 
having joint distribution P we have 

P{d(Z, V ) >e}^e. 

Since this topology has the second countability property, the metric space defined by the 
distance p(-, •) is separable. This can also be seen directly. Let S be a countable everywhere 
dense set of points in X. Consider the set of Mo(X) of those probability measures that 
are concentrated on finitely many points of S and assign rational values to them. It can 
be shown that Mq{X) is everywhere dense in the metric space (M(X),p); so, this space is 
separable. It can also be shown that (M(X),p) is complete. Thus, a measure can be given 
as the limit of a sequence of elements p,i,p2,... of M.q(X), where p(pi, Pi+i) < 2~\ 

The definition of the Prokhorov distance quantifies over all Borel sets. However, in an 
important simple case, it can be handled efficiently. 

Proposition B.17. Assume that measure v is concentrated on a finite set of points S C X. 
Then the condition p(u, p) < £ is equivalent to the finite set of conditions 

p(A £ ) > v{A) - e (B.7) 

for all AcS. 

B.6.3. Relative compactness. A set n of measures in (M(X),p) is called relatively compact if 
every sequence of elements of IT contains a convergent subsequence. Relative compactness 
is an important property for proving convergence of measures. It has a useful characteri- 
zation. A set of IT of measures is called tight if for every e there is a compact set K such 
that n(K) > 1 — e for all p, in LL Prokhorov's theorem states (under our assumptions of the 
separability and completeness of (X, d)) that a set of measures is relatively compact if and 
only if it is tight and if and only if its closure is compact in (M(X),p). In particular, the 
following fact is known. 

Proposition B.18. The space (M(X),p) of measures is compact if and only if the space (X, d) 
is compact. 

So, if (X, d) is not compact then the set of measures is not compact. But still, each 
measure /i is "almost" concentrated on a compact set. Indeed, the one-element set {/i} is 
compact and therefore by Prokhorov's theorem tight. Tightness says that for each e a mass 
of size 1 — e of n is concentrated on some compact set. 
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Appendix C. Computable analysis 

If for some finite or infinite sequences x, y, z, w, we have z = wxy then we write w C z 
Qw is a prefix of z) and x < z. For integers, we will use the toupling functions 

(hj) = \{i + !)(* + 3 + !) + 3, ( n i, ■ ■ -, n k+i) = ((n l5 . . . ,n k ),n k+1 ). 
Inverses: 7rf(n). 

Unless said otherwise, the alphabet £ is always assumed to contain the symbols and 1 . 
After [24], let us define the wrapping function i : S* — > S* by 

t(aia2 • • • a re ) = HOaiOc^O • • • a n 011. (C.l) 

Note that 

\l{x)\ = (2\x\ + 5) V6. (C.2) 

For strings x,^ G £*, p,pj G k ^ 1, appropriate tupling functions (xi, . . . ,x k ), (x,p), 
(p, x), etc. can be defined with the help of (■, •) and t(-). 

C.l. Notation and representation. The concepts of notation and representation, as de- 
fined in [24], allow us to transfer computability properties from some standard spaces to 
many others. Given a countable set C, a notation of C is a surjective partial mapping 
5 :C N — > C. Given some finite alphabet £ 5 {0, 1} and an arbitrary set S, a represen- 
tation of S 1 is a surjective mapping x S w — > S 1 . A naming system is a notation or a 
representation. Here are some standard naming systems: 

1. id, the identity over S* or 

2. fpj, v%, vq for the set of natural numbers, integers and rational numbers. 

3. Cf : S w — > 2 N , the characteristic function representation of sets of natural numbers, is 
defined by Cf(p) = { i : p(i) = 1 }. 

4. En : S 1 ^ — > 2 N , the enumeration representation of sets of natural numbers, is defined by 
En(p) = G X* : 110 n+1 ll < p }. 

5. For ACS, EnA : S w — > 2 A *, the enumeration representation of subsets of A*, is defined 
by EnA (jp) = {w G S* : l(w) <p}. 

One can define names for all computable functions between spaces that are Cartesian 
products of terms of the kind £* and Then, the notion of computability can be trans- 
ferred to other spaces as follows. Let 5i : Yi — > Xi, i = 1, be naming systems of the spaces 
Xi. Let / :C X\ — > Xq, g'.QY\—*YQ. We say that function g realizes function / if 

f(Si(v)) = SMv)) CC.3) 

holds for all y for which the left-hand side is defined. Realization of multi-argument func- 
tions is defined similarly. We say that a function / : X\ x X2 — > Xq is (<5i, 52 , 5o)-computable 
if there is a computable function 5 : C Y\ x I2 — > ^0 realizing it. In this case, a name for / 
is naturally derived from a name of g. 2 

For representations £,r], we write £ ^ 77 if there is a computable function / :C E u -t 
with = rj(f(x)). In words, we say that £ is reducible to 77, or that / reduces (translates) 
£ to rj. There is a similar definition of reduction for notations. We write £ = r/ if £ ^ 77 and 

C.2. Constructive topological space. 



Any function g realizing / via (C.3) automatically has a certain extensivity property: if Si(y) = <5i(j/') then 

g{y) = g(v')- 
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C.2.1. Definitions. Section A gives a review of topological concepts. A constructive topolog- 
ical space X = (X, a, v) is a topological space over a set X with a subbase a effectively 
given as a list a = {u(l),u(2), . . .}, and having the T property (thus, every point is deter- 
mined uniquely by the subset of elements of a containing it) . By definition, a constructive 
topological space satisfies the second countability axiom. 3 We obtain a base 

of the space X by taking all possible finite intersections of elements of a. It is easy to 
produce an effective enumeration for a n from v. We will denote this enumeration by v n . 
The product operation is defined over constructive topological spaces in the natural way. 

Examples C.l. 

1 . A discrete topological space, where the underlying set is finite or countably infinite, with 
a fixed enumeration. 

2. The real line, choosing the base to be the open intervals with rational endpoints with 
their natural enumeration. Product spaces can be formed to give the Euclidean plane a 
constructive topology. 

3. The real line R, with the subbase cr^ defined as the set of all open intervals (— oo; b) with 
rational endpoints b. The subbase a^, defined similarly, leads to another topology. These 
two topologies differ from each other and from the usual one on the real line, and they 
are not Hausdorff spaces. 

4. Let X be a set with a constructive discrete topology, and I u the set of infinite sequences 
with elements from X, with the product topology: a natural enumerated basis is also 
easy to define. 



Due to the To property, every point in our space is determined uniquely by the set of open 
sets containing it. Thus, there is a representation 7x of X defined as follows. We say that 
7x(p) = x if Ens(p) = { w : x E v{w) }. If 7x(p) = x then we say that the infinite sequence 
p is a complete name of x: it encodes all names of all subbase elements containing x. From 
now on, we will call 7x the complete standard representation of the space X. 4 

C.2.2. Constructive open sets, computable functions. In a constructive topological space 
X = (X,a,u), a set G C X is called r.e. open in set B if there is a r.e. set E with 
G = U w€E v n (w) n B. It is r.e. open if it is r.e. open in X. In the special kind of spaces 
in which randomness has been developed until now, constructive open sets have a nice 
characterization: 

Proposition C.2. Assume that the space X = (X, a, v) has the form Y\ x • • • x Y n where 
each Yi is either S* or T, w . Then a set G is r.e. open iff it is open and the set { (w\ , . . . , w n ) : 
Hi v(wi) C G} is recursively enumerable. 

Proof. The proof is not difficult, but it relies on the discrete nature of the space S* and on 
the fact that the space S w is compact and its base consists of sets that are open and closed 
at the same time. □ 



A constructive topological space is an effective topological space as defined in [24], but, for simplicity we 
require the notation v to be a total function. 

4 The book [24] denotes 7x as <5x instead. We use 7x only, dispensing with the notion of a "computable" 
topological space. 
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It is easy to see that if two sets are r.e. open then so is their union. The above remark 
implies that a space having the form Y\ x • • • x Y n where each Yi is either S* or Y!^ ', also the 
intersection of two recursively open sets is recursively open. We will see that this statement 
holds, more generally, in all computable metric spaces. 

Let X, = (Xj,<Tj,fi) be constructive topological spaces, and let / :C X\ — ► Xq be a 
function. As we know, / is continuous iff the inverse image f^ 1 (G) of each open set G is 
open. Computability is an effective version of continuity: it requires that the inverse image 
of subbase elements is uniformly constructively open. More precisely, / :C X\ — > Xq is 
computable if the set 

U r\v) x {v} 

is a r.e. open subset of X\ x Cq . Here the base ctq 1 of Xo is treated as a discrete construc- 
tive topological space, with its natural enumeration. This definition depends on the enu- 
merations vx,vq. The following theorem (taken from [24]) shows that this computability 
coincides with the one obtained by transfer via the representations 7x, . 

Proposition C.3. For i = 0, 1, let Xj = (Xi,Gi, Vj) be constructive topological spaces. Then a 
function f :C X\ — > Xo is computable iff it is (jx. 1 ,Tx. )-computable for the representations 
7x 4 defined above. 

As a name of a computable function, we can use the name of the enumeration algorithm 
derived from the definition of computability, or the name derivable using this representation 
theorem. 

Remark C.4. As in Proposition C.2, it would be nice to have the following statement, at least 
for total functions: "Function / : X\ — ► Xo is computable iff the set 

{(v,w):^(w)cr 1 [Mv)]} 

is recursively enumerable." But such a characterization seems to require compactness and 
possibly more. 

Let us call two spaces X\ and Xq effectively homeomorphic if there are computable maps 
between them that are inverses of each other. In the special case when Xq = X\, we say 
that the enumerations of subbases uq, v\ are equivalent if the identity mapping is a effective 
homeomorphism. This means that there are recursively enumerable sets F, G such that 

v i{v) = (J vq (w) for all v, vq(w) = vi(v) for all w. 

(v,w)eF (w,v)£G 

Lower semicomputability is a constructive version of lower semicontinuity. Let X = 
(X, a, v) be a constructive topological space. A function /:Cl-» M + is called lower semi- 
computable if the set { (x, r) : f(x) > r } is r.e. open. Let Y = a^, i/jg ) be the effective 
topological space introduced in Example C.1.2, in which is an enumeration of all open 
intervals of the form (r; oo] with rational r. It can be seen that / is lower semicomputable 
iff it is (u, z/^ - ) -computable. 

C.2. 3. Computable elements and sequences. Let U = ({0}, do, vq) be the one-element space 
turned into a trivial constructive topological space, and let X = (X, a, v) be another con- 
structive topological space. We say that an element x G X is computable if the function 
i— > x is computable. It is easy to see that this is equivalent to the requirement that the set 
{ u : x G v(u) } is recursively enumerable. Let X 7 = (Xj,aj, uj), for i = 0, 1 be constructive 
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topological spaces. A sequence fa, i = 1,2, ... of functions with f\ : X\ — > X is a com- 
putable sequence of computable functions if (i,x) i— ► is a computable function. Using 
the s-m-n theorem of recursion theory, it is easy to see that this statement is equivalent 
to the statement that there is a recursive function computing from each i a name for the 
computable function /j. The proof of the following statement is not difficult. 

Proposition C.5. Let X, = (Xi,Oi,Vi) for i = 1,2, be constructive topological spaces, and 
let f : X\ x X2 — > Xq, and assume that x\ G X\is a computable element. 

1. If f is computable and then X2 1— > f(x\,X2) is also computable. 

2. If Xo = M, and f is lower semicomputable then X2 f(xi,X2) is also lower semicom- 
putable. 

C.3. Computable metric space. Following [4], we define a computable metric space as a 
tuple X = (X, d,D,a) where (X, d) is a metric space, with a countable dense subset D and 
an enumeration a of D. It is assumed that the real function d(a(v),a(w)) is computable. 
As x runs through elements of D and r through positive rational numbers, we obtain the 
enumeration of a countable basis { B{x, r) : x G D,r G Q } (of balls or radius r and center 
x) of X, giving rise to a constructive topological space X. Let us call a sequence x\, X2, ■ ■ ■ a 
Cauchy sequence if for all i < j we have d(xi,Xj) < 2~\ To connect to the type-2 theory of 
computability developed above, the Cauchy -representation 5x of the space can be defined 
in a natural way. It can be shown that as a representation of X, it is equivalent to 7 X : 
<*x = 7x- 

Example C.6. Example A.5 is a computable metric space, with either of the two (equivalent) 
choices for an enumerated dense set. 

Similarly to the definition of a computable sequence of computable functions in C.2.3, 
we can define the notion of a computable sequence of bounded computable functions, or 
the computable sequence fa of computable Lipschitz functions: the bound and the Lipschitz 
constant of f are required to be computable from i. The following statement shows, in an 
effective form, that a function is lower semicomputable if and only if it is the supremum of 
a computable sequence of computable functions. 

Proposition C.7. Let X be a computable metric space. There is a computable mapping that to 
each name of a nonnegative lower semicomputable function f assigns a name of a computable 
sequence of computable bounded Lipschitz functions f i whose supremum is f. 

Proof sketch. Show that / is the supremum of a computable sequence of computable func- 
tions CilB(ui,n) where u\ G D and Cj,r; > are rational. Clearly, each indicator function 
^B(ui,ri) is tne supremum of a computable sequence of computable functions gij. We have 
/ = sup n f n where /„ = maxj^ n Cig^ n . It is easy to see that the bounds on the functions 
/„ are computable from n and that they all are in Lip« for a (3 n that is computable from 
n. □ 

The following is also worth noting. 

Proposition C.8. In a computable metric space, the intersection of two r.e. open sets is r.e. open. 

Proof. Let = {B(x,r) : 1 e D,r 6 Q} be a basis of our space. For a pair (x,r) with 
x G D, r € Q, let 



F(x, r) = {(y,s) : y G D, s G Q, d(x, y) + s < r }. 
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If U is a r.e. open set, then there is a r.e. set Sjj C D x Q with U = \J, X r ^ e g v B(x, r). Let 
S'y = U{ T(x, r) : (x, r) e Sjj }, then we have U = \J^ X r ) g s^ B(x, r). Now, it is easy to see 

UnV= \J B(x,r). 

□ 
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